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ABSTRACT 


This  study  is  concerned  with,  the  stability  of  random  systems, 
that  is,  systems  whose  internal  characteristics  are  governed  by  prob¬ 
ability  laws. 

Concepts  of  stability  appropriate  to  random  differential  systems 
are  formulated  and  discussed.  Precise  definitions  of  stability  are 
stated  and  theorems  interrelating  these  definitions  are  proven.  Some 
particular  types  of  stability  investigated  are,  in  the  mean  norm,  in 
the  ith  moment,  in  probability,  almost  sure,  and  almost  uniform-in-  u. 

.  Particid.ar  attention  is  focused  on  the  random  linear  (vector)  differ¬ 
ential  equation  with  piecewise  constant  parameters: 

3c  =  Aj^x,  ^  ^  ~  • 

Two  statistical  structures  ar,e  studied: 

^)}  is  an  independent,  identically  distributed, 
random  process; 

(b)  |))  ^  finite  Markov  chain. 

The  ith  moment  of  the  solutions  iy  obtained  explicitly  and  is  studied 
with  respect  to  asymptotic  behavior,  and  with  respect  to  the  set  struc¬ 
ture  of  the  Markov  chain  in  case  (b)*.  Sufficient  conditions  for  exponen¬ 
tial  asymptotic  stability  in  the  ith  moment  are  derived.  (In  case  (a), 
these  conditions  are  also  necessary.  )  It  is  shown  that  if  i  is  even,  then 
these  sufficient  conditions  also  imply  almost  sure  asymptotic  stability. 

For  random  linear  systems,  it  is  shown  that  if  the  trivial  solution 
of  the  homogeneous  equation  is  exponentially  asymptotically  stable  in 
tile  mean  norm,  then  certain  stochastic  equivalents  of  "bounded  output 
to  every  bounded  input"  are  true. 

With  minor  modifications,  all  of  the  above  results  are  equally 
applicable  to  random  difference  equations. 


<»> 


iii 


TABLE  OF  CONTENTS 


Acknowledgment  .  . .  ii 

Abstract . iii 

I.  Introduction  and  Summary  . .  1 

1. 1  General  Introduction  .  1 

1.  2  Some  Previous  Work  on  the  Stability  of  Random 

Systems .  2 

1,  3  Outline  of  Present  Study .  4 

II.  Stability  of  Random  Systems .  5 

Summary .  5 

2. 1  Deterministic  Systems .  5 

2.2  Random  Systems .  8 

Asymptotic  Considerations . 14 

•f 

ill.  Linear  Systems  with  Independent  Parameters . 17 

Summary  ..  . . , . 17 

3. 1  ith  Moment  Stability  .  . . 18 

3.  2  Almost  Sure  Asymptotic  Stability . 27 

3.  S  Example  1 . 29 

3,  4  Example  2 . 30 

IV.  Linear  Syateum  with  Markov.  Parameters . 33 

4: 1  Markov  Chains . 33 

4.  2  ith  Moment  Stability . 37 

^Dependence  on  Structure  of  Markov  Chain . ^41 

4.  3  Almost  Sure  Asymptotic  Stability . 47 

4.  4  Example  . .47 

4.  5  Some  General  Comments . 50 

V.'  The  Forced  Linear  System . 51 

Summary . 51 


Appendix  A:  Kronecker  and  Power  Products  of  Matrices . 56 

Kronecker  Products . 56 

Power  Products . 59 

Appendix  B:  A  Modified  Holder  Inequality . 60 

Appendix  C:  Some  Theorems  on  Almost  Sure  aind  Almost  Uniform- 

in-  (i)  Types  of  Stability . 61 

Appendix  D:  Matrix  Valued  Markov  Chain  .  .  64 

Appendix  E:  Specializaaion  of  Markov  Case  to  Independent  Case . 67 

T 

Appendix  F:  Computation  of  Cyclic  Chains . 69 

References  . 73 


IV 


1.  INTRODUCTION  AND  SUMMARY 


1. 1  General  Introduction 

By  a  random  system  one  understands  here  a  system  some  or  all 
of  whose  internal  characteristics  are  governed  by  probability  laws. 
More  precisely,  it  is  asstuned  that  the  unforced  system  can  be  repre¬ 
sented  by  an  ensemble  of  equations  indexed  by  the  element  w  of  a 
nrobability  space.  Thus,  for  example,  the  system  may  have  a  repre¬ 
sentation  as  a  linear  differential  equation  with  random  processes  as 
coefficients.  This  work  is  concerned  with  the  stability  of  systems  de¬ 
scribed  by  random  differential  or  difference  equations.  The  problems 
of  measurement,  identification  and  optimization  are  not  touched  upon. 

A  variety  of  physical  situations  can  be  represented  by  random 
differential  equations.  Two  examples  in  the  field  of  systems  are: 

(i)  an  adaptive  system  which  compensates  for  effects  of  an  external 
random  disturbance;  and  (ii)  a  control  system  whose  parameters  are 
undergoing  noise  modulation.  An  illustration  of  a  random  parameter 
is  the  gain  of  the  control  surface  of  an  aircraft  in  flight  through  a 
medixim,  possibly  turbulent,  of  rapidly  changing  and  incompletely 
known  characteristics.  In  the  field  of  circuits,  lumped  networks  con- 
taixiing  randomly  varying  elements  can  be  described  by  random  differ¬ 
ential  equations;  likewise,  long  transmission  lines  with  characteristics 
that  are  nonuniform  along  the  length  of  the  line.  ^  Finally,  problems 
of  wave  propagation  and  scattering  in  a  random  medium  also  lend  them¬ 
selves  to  a  random  differential  equation  representation.  ^ 

Random  difference  equations  arise  in  most  of  the  above  situations 

if  the  coefficients  of  the  equations  are  rajudom  but  piecewise  constant 

3  4 

in  the  independent  variable.  '  They  occur  also  in  the  field  of  quan- 

5 

turn  mechanics  in  the  study  of  energy  levels  in  random  lattices.  A 
final  example  is  the  area  of  randomly  sampled  systems,  that  is, 
sampled-data  systems  whose  sampling  intervals  are  random  vari- 
ables.  The  random  sampling  may  result,  for  instance,  from 
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economic  considerations,  as  in  the  time  sharing  of  a  digital  computer 
that  controls  several  processes;  from  requirements  of  secrecy  and 
anti-jam  protection,  as  in  military  communication  systems;  from  im- 
avoidable  perturbations  on  the  nominal  period  of  a  "periodically" 
sampled  system.  Finally,  there  are  situations  which  though  not 
involving  random  sampling  per  se,  can  be  formulated  in  terms  of  a 
sampler  which  "skips"  or  fails  to  operate  in  some  random  manner. 

Two  such  examples  are  data  transmission  links  and  scan  radars, 
where  the  received  data  may  be  rejected  at  some  time  instants  be¬ 
cause  of  excessive  noise. 

1.  2  Some  Previous  Work  on  the  Stability  of  Random  Systems 

For  random  linear  systems  with  continuous  parameter  variations, 

'  7 

some  results  are  available.  For  first-order  systems  Rosenbloom 

has  expressed  the  output  moments  in  terms  of  the  characteristic  f\mc- 

tion  of  the  indefinite  time  integral  of  the  parameter  process;  he  shows 

in  particular  that  if  the  parameter  process  is  stationary  and  Gaussian, 

then  for  a  step  input  the  first  and  second  output  moments  may  become 

tinbounded,  whereas  the  output  approaches  one  in  probability,  as  t-Mo. 

8 

Tikhonov  has  considered  the  first-order  case  where  the  input  and 
parameter  processes  are  jointly  Gaussian.  The  random  linear  differ¬ 
ential  system  containing  one  purely  random  coefficient  has  been  stud- 

1  9 

ied  in  some  detail  by  Samuels  and  Eringen.  Samuels  has  further 
developed  the  theory  and  extended  it  to  systems  containing  one  narrow- 
band  random  parameter.  He  has  considered  also  linear  systems  with 
dependent  parameter  processes  and  an  independent  input  and  has  ar¬ 
rived  at  specific  results  in  some  special  cases.  Bergen^^  has  also 
studied  the  linear  differential  system  containing  only  one  purely  ran¬ 
dom  coefficient,  and  has  found  necessary  and  sufficient  conditions 

for  the  mean  square  error  to  remain  bounded  when  the  system  is 

11  12 

excited  by  an  independent  input,  Zadeh  '  has  investigated  a  very 
general  class  of  random  linear  systems  admitting  of  a  certain  inte¬ 
gral  representation,  and  has  demonstrated  an  integral  relation 


between  the  output  corarlnnce,  and  the  input  and  "syatem"  corariance, 

eimilar  to  that  fox  determlnifftic  linear  ayoteme. 

13 

Bertram  and  Sarachik  hare  extended  "Lyapunor'e  Second  Method" 
to  random  lystema.  By  working  with  the  expectation  of  the  total  time 
deriratire  of  the  Lyapunor  function  along  the  system  trajectories  and 
measuring  stability  in  the  sense  of  the  mean  norm,  they  hare  arrired 
at  theorems  analogous  to  those  in  the  deterministic  case. 

For  discrete  random  linear  systems  (and  continuous  systems  with 
piecewise  constant  parameters),  the  problem  consists  of  studying  the 
beharior  of  a  giren  initial  state  rector  undergoing  a  succession  of  ran¬ 
dom  linear  transformations.  Kalman^  derired  the  necessary  and  suf¬ 
ficient  conditions  for  mean  square  stability  of  an  nth- order  randomly 
sampled  syatem  whose  sampling  interrals  form  a  sequence  of  indepen¬ 
dent,  identically  distributed,  random  rariables.  For  the  random  linear 

nth-order  system  with  piecewise  constant  parameters: 

*  3 

X  s  -A^x,  tj^_^  <  t  <  t^,  k  =  1,  . . .  ,  Bergen  has  found  necessary  and 

sufficient  conditions  for  asymptotic  stability  in  the  second  moment  when 
{A^)  1.  .  d.t.rminl«ic  ..que>ee  of  th.  form  {B.  C,  B,  C. . . .  )  m.d 

is  an  independent,  identically  distributed,  random  sequence. 
This  author^  has  inrestigated  the  asymptotic  stability  in  the  ith  moment 
for  the  case  where  ^  finite  Markor  chain  (see  Chaps. 

Ill,  lY).  All  three  abore  authors  use  the  derice  of  the  Kronecker  pro¬ 
duct  of  matrices,  first  used  by  Bellman^^  in  studying  the  asymptotic 
beharior  of  products  of  independent  random  linear  transformations. 

In  a  recent  paper  on  products  of  random  matrices,  Furstenberg  and 
Kesten^^  hare  found  general  conditions  for  the  conrergence  of  the  ran¬ 
dom  sequence  {n**^  log  (]  X^. . .  }  with  {X^}  a  stationary  sto¬ 

chastic  process  with  ralues  in  the  set  of  k  x  k  matrices,  and  hare  de¬ 
duced  the  asymptotic  normality  of  log(X  X  ....  X,},  a  result  earlier 

n  n-i  i 

conjectured  by  Bellman. 
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1.  3  Outline  of  Preeent  Study 

In  general,  tke  problem  of  etabillty  it  to  determine  the  extent 
to  wbiek  a  let  of  proportion  of  a  eyetem  remaino  invariant  under  a 
apecified  eet  of  dieturbancee  or  alteration!  on  the  system,  on  the 
initial  conditions,  etc.  For  deterministic  systems,  a  common  sta¬ 
bility  problem  la  the  study  of  deviations  of  the  system  state  vector 
from  a  given  equilibrium  state  (  or  equivalently  from  a  given  solution) 
when  Initial  conditions  are  close  to  this  equilibrium  state;  the  various 
stability  concepts  are  generated  by  imposing  requirements  that  the 
magnitude  of  the  deviations  remain  small  for  all  time,  that  it  returns 
to  aero  as  t->ao,  that  it  satisfies  uniformity  conditions  with  respect 
to  initial  conditions,  etc.  The  corresponding  problem  for  random  sys¬ 
tems  is  considered  here,  the  deviation  now  being  measured  in  some 
stochastic  sense,  e.  g. ,  in  probability,  almost  surely  (with  probabil¬ 
ity  one),  almost  tmifermly  (-in-  u),  in  the  ith  mean,  etc. 

In  Chap.  II,  concepts  of  stability  appropriate  to  random  differ¬ 
ential  systems  are  formulated  and  discussed.  Precise  definitions  of 
stability  are  given  and  theorems  interrelating  these  definitions  are 
proven.  In  the  beginning  of  the  chapter,  numerous  definitions  of  sta¬ 
bility  are  stated  to  provide  the  motivation  for  the  definitions  in  tits 
random  case.  Although  here,  and  in  the  remaining  chapters,  the  sys¬ 
tem  is  assumed  to  be  a  differential  one,  by  making  small  changes  it 
is  possible  to  apply  the  material  presented  to  other  types  of  systems, 
e.  g. ,  to  systems  of  random  difference  equations. 

Chapter  Ill  is  devoted  to  the  stability  analysis  of  the  random  lin¬ 
ear  difCoreatial  system  with  piecewise  constant  coefficients: 

jt  ®  Aj^x,  ^  ^  ^  k  —  I,  2,  . . .  , 

Where  x(t)  is  an  n-vector  and  A^^  is  a  constant  n  x  n  matrix.  The  be¬ 
havior  of  the  system  is  completely  determined  by  the  random  sequence 
■’tj^^^)},  aesumed  in  Chap  .  Ill  to  be  an  indepondent,  identically 
distributed,  random  process.  Seme  particular  results  obtained  are 
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the  necessary  and  sufficient  conditions  for  asymptotic  stability  in  the 
ith  moment  and  sufficient  conditions  for  almost  sure  asymptotic  sta¬ 
bility. 

Chapter  IV  considers  the  same  system  as  Chap.  Ill  except  that 
{Aj^(tk“tk_i)l  is  now  a  finite  Markov  chain.  The  results  obtained  are 
almost  identical  to  those  of  Cha^iIU.  In  fact,  all  the  results  of  Chap.  Ill 
can  be  deduced  from  those  of  Chap. IV  (and  Appendix  E).  The  indepen¬ 
dent  case  is  treated  separately  in  Qiap.III  because  it  seems  preferable 
to  do  so  from  an  expository  point  of  view. 

The  preceding  chapters  have  considered  the  unforced  system.  In 
practice,  however,  it  is  the  behavior  of  the  system  in  the  presence  of 
an  input  that  is  of  interest.  In  Chap.  V,  a  theorem  is  proven  for  random 
linear  systems  which  shows  that,  if  the  unforced  system  is  exponen¬ 
tially  asymptotically  stable  in  the  mean  norm,  then  some  stochastic 
equivalents  of  "bounded  output  to  every  bounded  input"  are  true. 

II.  STABILITY  OF  RANDOM  SYSTEMS 

Summary 

Concepts  of  stability  Appropriate  to  random  differential  systems 
are  formulated  and  discussed.  Precise  definitions  of  stability  are  given 
and  theorems  interrelating  these  definitions  are  proven.  In  the  beginning 
of  the  chapter  numerous  definitions  of  stability  of  deterministic  differ¬ 
ential  systems  are  stated  to  provide  the  motivation  for  the  definitions 
in  the  random  case.  Although  the  system  is  assumed  to  be  a  differential 
one,  by  making  small  changes  it  is  possible  to  apply  the  material  pre¬ 
sented  here  to  other  types  of  systems,  e.  g.  ,  systems  of  random  differ¬ 
ence  equations. 

2.  1  Deterministic  Systems 

Consider  the  differential  equation 

4 

i  =  f(x,t)  (2.1) 

^  where  x  is  an  element  of  the  real,  (normed)  vector  n- space  R°^,  and  f 
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ii  defined  on  r“x  {t;t  The  function  f  it  attumed  to  be  suffi¬ 

ciently  smooth  to  that  solutions  of  (2. 1)  exist  and  are  unique  for 
all  t  >  0  for  all  initial  values  =  x(tQ),  t^  >  0,  and  are  continuous 
35 

^  ^*0’  *0^*  *0'  *0^  designate  that  solution  which 

satisfies  the  initial  condition  git^,  x^,  t^)  =  x^.  By  an  abuse  of  nototion. 
x(t)  also  will  be  used  to  designate  a  solution  but  the  context  will  be  such 
at  to  preclude  any  ambiguity  with  the  variable  x  of  the  differential  equa¬ 
tion  (2. 1). 

Without  loss  of  generality  it  is  attumed  that  f(0,  t)  =  0  for  all  t  >  0 
so  that  X  ■  0  is  always  a  solution,  called  the  trivial  solution.  For,  given 
any  solution  of  a  differential  system,  it  is  always  possible  by  introduc¬ 
ing  a  change  of  variables  in  the  differential  equations  to  arrive  at  a  new 
differential  system^^  which  satisfies  the  relation  f(0,  t)  =  0  for  all  t  >  0. 

The  term  stability  as  considered  here  is  the  study  of  the  deviation, 
from  the  trivial  solution,  of  solutions  corresponding  to  nonxero 
conditions. 

DEFINITIONS  *:  The  trivial  solution  g(t,  0,  t^)  =  0  is  called 

^  given  «  >  0,  tp,  there  exists  >  0  such  that 

II  *0 II  ®  implies  }j  g{t,  Xq,  t^)  |  <  t  for  all  t  >  t^. 

{ii)  uniformly  stable  if  given  «  >  0,  there  exists  6(c)  >0  such 
that  for  all  tg,  ||xq|I  <6  implies  jj  g(t,XQ,tp)l|  <c  forallt>tQ. 

(iii)  quasi-asymptotically  stable  if  given  tp,  there  exists 
6(tp)  >  0  such  that  II  Xpl  <6  implies  g(t,  Xp,tp)-p0  as  t  — »oo. 

(iv)  asymptotically  stable  if  it  is  stable  and  quasi-asymptotically 
stable. 

(v)  quasi- equiasymptotically  stable  if  given  any  tp,  there  exists 
fi(to)  >  0  such  that  |  Xpj|  <  6  implies  g(t,  Xp,tp)— ♦  0  as  t— »oo 
uniformly  on  |  Xp  |  <6. 

V 

It  is  understood  that  t  _^tp  >  0. 
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(Vi)  equiasymptotically  stable  if  it  is  stable  and  quasi-equia- 
symptotically  stable. 

(vii)  quasi- uniformly  asymptotically  stable  if  there  exists 

6  >  0  such  that  for  all  t^,  U  x^U  <6  implies  g(t,  x^,  t^) - ►  0  as 

t - >  00  uniformly  on  t^  >  0,  ||  x^  ||  <  6. 

(viii)  uniformly  asymptotically  stable  if  it  is  uniformly  stable 
and  quasi-uniformly  asymptotically  stable. 

(ix)  exponentially  asymptotically  stable  if  there  exists  a  v  >0, 
and  given  any  s  >  0,  there  is  a  corresponding  6(c)  >  0  such  that  for 
all  tjj,  IIxqU  <  6  implies  ||  g(t,  x^.t^)]!  <  c  exp  [  -  v  (t-t^)]  for  all 

t>V 

Implications  such  as  the  following  ones  are  obvious: 

(ii)  =»  (i) 

(vii)  ==^  (v)  >  (iii) 

(viii)^=^(i)  -  (vii) 

For  other  implications,  examples,  and  discussion  of  the  definitions,  see 

17  18  19 

Antosiewicz,  Massera,  Kalman  and  Bertraun.  The  latter  two 

references  also  consider  concepts  of  stability  in-the-large. 

If  the  system  is  linear,  that  is  if  f(x,  t)  is  a  linear  in  x,  then  (viii) 

aind  (ix)  are  equivalent;  in  fact,  for  (ix)  there  is  a  constant  K  >  1  such 

that  it  is  possible  to  choose  6(c  )  =  e  /  K  so  that 

II  g(t,XQ,tQ)l|  <K||xQ|j  exp[  -  v(t-tQ)]  for  all  t  >  tp.  See  Massera. 

Definition  (i)  requires  that  the  trivial  solution  be  stable  for  every 
initial  time  t^.  Since  the  solution  functions  are  assumed  to  be  continu¬ 
ous  in  the  initial  value  x^  it  suffices  to  have  the  trivial  solution  stable 
for  some  t^;  namely  the  trivial  solution  is  stable  if  (and  only  if)  there 
exists  some  number  t^  with  the  property  that  given  any  c  >  0  there  is 
a  6(€  ,  tjj)  >  0  such  that  ||  Xq||  <6  implies  ||  g(t,XQ,tQ)||  <e  for  all 
t  >  Iq.  See  Kalman  and  Bertram. 

Similarly  for  definition  (iii),  it  suffices  to  have  quasi-asymptotic 
stability  for  some  initial  time  t^;  that  is  in  order  that  the  trivial  solu¬ 
tion  be  quasi-asymptotically  stable,  it  suffices  to  have  some  t^  and  a 
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corresponding  6(1^)  >  0  such  that  1|  Xq||  <6  implies  g(t,  x^,  t^) - >0  as 

t — >00.  For,  given  any  other  initial  time  1^,  consider  the  mapping  of 
the  set  Eq  =  {xqI  ||  Xq||  <  6(tQ)  }  under  the  solution  function  to  the  set 
*Q»  ^q)*  II  *0 II  ^  ^  Since  the  solutions  are  unique 

and  continuous  in  the  initial  state,  the  mapping  Eq— >  EJj  is  a  homeomor- 
phism.  Further,  E|j  contains  the  point  xjj=  0  for  this  is  the  map  of  the 
point  Xq  >  0.  Hence,  there  is  a  set  Eg,  =  {  x|j :  xjj  <  6'}  for  some 

6'  >  0,  and  g(tQ,  x^  ,  t^ )  a  Eq  for  every  xjj  e  Eg,  ,  which  implies  that 
g(t,  xjj ,  t|j ) — >0  as  t  — >  00. 

All  of  the  above  stability  properties  are  independent  of  the  choice  of 
the  norm.  For,  given  any  two  norms  ||  •  and  ||  •  ||  defined  for 

the  elements  x  of  a  finite-dimensional  vector  space,  there  exist  two  con¬ 
stants  kj2,  (see  Householder^*^,  Naimark^^). 

>'2lll«ll‘^’  <  for  all,, 

2.  2  Random  Systems 

The  differential  equation  (2. 1)  now  becomes 

x±f(x,t;w)  (2.2) 

where  w  is  a  point  in  a  probability  space  n.  For  every  fixed  w,  f  satisfies 
the  assumptions  made  at  the  beginning  of  the  previous  section  on  deter¬ 
ministic  systems.  The  time  interval  of  definition  of  f,  [  0,  oo)^  is  fixed, 
i.  e.  ,  does  not  depend  on  the  choice  of  w.  Also,  f(0,t;u)  =  0  on  [0,  oo) 

Xn  so  that  again  x  =  0  is  always  a  solution. 

To  every  (x^.t^)  there  corresponds  the  set  of  solutions  {g(t,  Xq,  tQ;w), 
w«J2  }  with  g(tQ,  Xq,  tQ;«)  =  x^  for  all  w;  that  is,  to  every  (x^,  t^)  there 
corresponds  a  random  process,  namely, a  family  of  random  variables 
indexed  by  t.  The  problem  of  stability  is  that  of  studying  the  probabi¬ 
listic  departure,  from  the  trivial  solution,  of  the  class  of  random  pro¬ 
cesses  generated  when  x^  is  taken  to  be  different  from  zero.  The  defi¬ 
nitions  given  in  the  previous  section  are  to  be  modified  So  as  to  take 
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into  account  the  dependence  on  w  of  the  solution  function  g(t,XQ,tQ:  u). 
This  dependence  on  w,  however,  will  not  be  exhibited  always  by  the 
notation  in  most  cases  solutions  will  be  designated  by  g(t,XQ,  t^). 
Thus  from  the  previous  definition  of  stable,  one  may  obtain 
DEFINITION  (i-a):  The  trivial  solution  g(t,  0,  t^)  =  0  is  called 
stable  in  the  mean  norm  if  given  s  >  0,  t^,  there  exists  6  (c ,  tQ)  >  0 
such  that  II  Xq  II  <6  implies  E  ||  g(t,  x^,  t^)  ||  <  c  for  all  t  >  t^. 

In  coilnectlon  with  the  above  definition  it  is  useful  to  note  that 

(1)  Norms  such  as 

II  *11  p  =  (  ^  I  *i!^)  .  P  >1  (2.  3) 

may  not  be  desirable  from  the  viewpoint  of  analysis  for  they  involve 
the  operations  of  taking  the  absolute  value  and  extracting  the  pth  root. 
However,  for  any  scalar  random  variable 

r  1/r,  r,  l/r^ 

(E|z|  ^  <(E|z|  ,  0<t^<T2  (2.4) 

Then  for  any  fixed  P  >  1,  it  follows,  by  setting  Z  =  |  x J  ^  , 

r^  =  l/p,  and  =  1,  that 

(2.5) 

Hence,  in  establishing  the  property  of  stable  in  the'meam  norm  with 
respect  to  the  norm  ||  x|  it  suffices  to  show  that  £  [  g  J  ^  < 

rather  than  E||  g||  ^  <  e  as  in  definition  (i).  The  quantity  ||  x||  ^ 
is  not  endowed  with  the  homogeneity  property  of  the  norm , 

II  ax||  =  I  a|  II  xU  ,  on  which  rests  the  invariamce  property: 

(2)  As  in  the  deterministic  case,  the  choice  of  the  particular 
norm  is  irrelevant  --  the  property  of  stability  in  the  mean  norm  is 
invariant  under  changes  of  the  norm. 

It  follows  from  (1)  and  (2)  that  stability  in  the  mean  norm  is  a 
consequence  of  mean  square  stability,  defined  as: 
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DEFINITION  (i-b^:  The  trivial  solution  is  called  mean  square 

stable  if  to  every  <  >  0,  t^,  there  corresponds  a  6(c  ,  t^)  >  0  such  that 
||xq11  <  6  implies 

^  ^  (*i^^'  *0’  ^0^)^  ^  ^  -  *0' 

i=l 


KRONECKER  AND  POWER  PRODUCTS.  The  theory  of  Kronecker 
and  power  products  of  matrices  is  connected  with  the  problem  of  sta¬ 
bility  in  the  mean  square  sense,  or  more  generally,  in  the  sense  of  the 
ith  moment,  of  linear  systems.  Appendix  A  defines  the  Kronecker  and 
the  power  products  and  states  some  of  their  properties.  Stability  con¬ 
cepts  based  on  the  Kronecker  and  the  power  product  are  motivated  and 
developed  in  ChapalU  and  IV.  The  brief  discussion  below  is  included 
here  for  the  sake  of  completeness. 

The  Kronecker  product  of  the  n-vector  x  =  (  ...  C  )  by  the 

m-vector  y  =  (  q,,  q.,  . .  .  ,  q  )  is  the  nm-vector 

X  ®  y  =  (Si’ll.  5i»l2.  •  •  •  .  5iV'  ^2^1’  ^2^2’  •  •  *  ' 


having  as  components  all  possible  products  of  a  component  of  x  by  a 
component  of  y.  The  Kronecker  product  of  x  by  itself  is  the  n-vector 
X  X  and  is  denoted  by  x^  The  ith  self-Kronecker  product  may 
be  defined  by  the  recursive  relation: 

‘[1]  ‘  = 

^~X[i  1]  ®  ^  since  the  product  is  associative).  The  vector  x^^j  is 
taken  fb>be  identically  x  . 

The  power  product  of  x  by  itself  is  the  n(n  +1)  /  2  vector 
*(2)  =  ^Ml’  M2’  Mn'  ^2^2’  ^2^3’  •  ^nV 


For  an  nth-order  scalar  differential  equation,  the  term  mean 
square  stable  is  vised  often  in  the  sense  that  the  .expectation  of  the  square 
of  the  (scalar)  solution  function  remains  small.  See  Refs.  9  and  10. 
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having  as  components  distinct  products  of  components  of  x.  A  higher 
power  is  similarly  defined  as  the  vector  having  as  components  the 

^n+i-l^  distinct  ith-order  products  of  components  of  x.  The  lexico¬ 
graphic  ordering  is  a  possible  ordering  for  the  components  of  x^^^,  and  is 
the  one  obtained  from  the  Kronecker  product  x^^j  by  inspecting  successive¬ 
ly  the  components  of  x^  starting  with  the  first  component  and  deleting 

any  component  which  is  identical  to  any  preceding  component.  Thus,  if 

2  2  2  2 
x  =  (u,  v),  thenxj2j  =  (u  ,uv,vu,  v  ),  and  x^2)  =  . 'iv,  v  ). 


Returning  to  the  discussion  of  stability, 


DEFINITION  (i-c):  The  trivial  solvdion  is  said  to  be  stable  ip.  the 
ith  moment  if  given  any  c  >  0,  t^^  there  exists  6(  c  ,  t^)  >  0  such  that 
|xq|  <6  implies(|  Egjg(t,XQ,tQ)||  <€  for  all  t  >  t^. 

The  above  definition  remains  unchanged  if  the  Kronecker  product 
g^j.j  is  replaced  by  the  power  product  8^2)*  replacement  will  net, 

in  general,  alter  in  any  significant  way  most  of  the  statements  made  in 
this  report.  It  turns  out  that  for  theoretical  analyses  it  is  simpler  to 
use  the  Kronecker  product  for  one  is  not  plagued  by  considerations  of 
ordering  of  the  components  of  the  vectors  and  of  the  matrices.  Hence, 
the  Kronecker  product  will  be  used  exclusively  in  the  sequel.  However, 
when  numerical  computations  have  to  be  performed  in  a  particular  case, 
the  problem  can  be  formulated  in  terms  of  the  power  product  for  the 
smaller  order  of  the  power  product  vector  can  be  of  distinct  computa¬ 
tional  advantage. 

It  is  apparent  from  the  definition  of  the  Kronecker  product  and  from 
definition  (i-c)  that  stability  in  the  ith  moment  requires  not  only  that  the 
ith  moments  of  the  individual  components  of  g  be  small,  but  also  that 
all  ith-degree  cross  moments  of  components  of  g  be  small.  The  value 
•f  this  latter  requirement  on  the  cross  moments  becomes  clear  in 
ChapsiU  ud  IV.  But  it  should  be  mentioned  here  that  if  i  is  even,  then 
the  requirement  on  the  cross  moments  is,  in  a  sense,  superfluous. 

This  follows  from  the  inequality 
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(2.6) 


i.  i. 

ETr|z.J|  <7Te|.z.j|  J 

j  J  "  j  J 


I  ‘j  =  ‘.  lj>0. 
j 


j  =  1,  2,  •  •  •  ,  n 


where  the  Zj  arc  scalar  random  variables.  The  ineqviallty  is  a  modified 
form  of  the  Holder  inequality  and  is  proven  in  Appendix  B.  Hence,  for 
i  even,  if  for  the  vector  g  =  (g^,  ,  g^),  Egj  is  small  for  all  J, 


then  so  is 


4  4 

=<*1 «? 


). 


Consequently,  definition 


(i-c)  of  stable  in  the  ith  moment  for  even  i  remains  unchanged 

n 

if  II  II  1*  replaced  by  E  ^  gj  .  In  particular,  stability  in  the 

j=l 

second  moment,  (i-c),  is  equivalent  to  mean  square  stability,  (i-b). 
Moreover,  by  inequality  (2.4),  it  follows  that  if  i  is  even  then  stability 
in  the  ith  moment  Implies  stability  in  the  Jth  moment  for  all  even  j  <  1. 
Hence,  it  also  follows  that  stability  in  the  ith  moment  for  any  even  i 
implies  stability  in  the  mean  norm  (irrespective  of  the  norm).  The 
various  properties  established  above  can  be  of  value  in  the  stability 
analysis  of  a  given  system  where  the  indiviiVial  nature  of  the  problem 
may  make  it  more  pertinent  or  expedient  to  consider  a  particular  type, 
rather  than  some  other  type,  of  stability. 

Unlike  the  three  preceding  definitions,  the  two  following  definitions 
do  net  involve  any  "averaging"  operations. 


DEFINITION:  The  trivUl  solution  is  called 

(i-d)  stable  in  probability  if  given  c  >0,  q  >  0,  t^,  there  exists 
«(  <  ,  q#  ^  °  B  *0^  ^  *  implie*  Pf  i  g(t,  x^,  tQ)||  <•  ]>l-q 

for  all  t  >  tg. 
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(i-e)  atable  «lmo«t  jurely  If  there  exists  an  u-set  S  with  P(S)  =  1, 
and  given  u  t  S,  c  >0,  t^,  there  exists  6^(«  >  tQ)  >  0  such  that  for 
all  wsS,  IIxqI  <6^  implies  |1  g(t, x^, tQ;w) |  <c  forallt>tQ; 
that  is,  for  almost  every  la,  the  trivial  solution  is  stable. 

The  dependence  of  i^onw  is  somewhat  unsatisfactory  for  given  t^  and  a 

bound  c  on  the  solutions,  there  is  no  guarantee  that  one  can  find  a  common 

bound  fi(c  ,  t^)  on  the  Initial  values  x^  to  insure  that  the  norm  of  almost 

all  realizations  (i.  e. ,  for  all  u  belonging  to  some  set  S  with  P(S)  =  1) 

remains  within  <  .  Clearly  =  inf  6  is  the  largest  possible  6; 

wsS  “ 

if_6  s  0,  however,  then  no  such  6  exists.  In  general,  for  the  systenis 
considered  in  the  following  sections,  6=0  and  this  property  of  uniform^ 
ity  oyer  a  set  of  probability  1  will  not  be  discussed  further.  If,  however, 
it  suffices  to  have  a  set  of  probability  arbitrarily  close  to,  but  not  neces¬ 
sarily  equal  to,  one,  on  which  there  is  a  uniform  bound  on  the  initial  con¬ 
ditions,  then  it  can  be  shown  that  this  is  equivalent  to  almost  sure  stabi¬ 
lity.  This  motivates  the  following*. 

DEFINITION  (i-f):  The  trivial  solution  is  called  stable  almost  uni¬ 
formly -in- m  if  given  any  >  0,  t^,  there  exists  an  u-set  B(q,tQ)  with 
P(B)  >  1  -  T]  and  given  any  c  >  0,  t^,  there  corresponds  a  number 

>0  such  that  |  x^U  <6  implies  |1  g(t,  Xq,  t^Jw)  |  <e  for  all 
t  >  tg  and  all  w  s  B. 

THEOREM  1:  The  trivial  solution  of  (2.  Z)  is  stable  almost  surely 
if  and  only  if  it  is  stable  almost  uniformly-in-u.  (Proof:  See  Appendix 
C.) 

It  is  clear  that  (i-f)  implies  (i-c);  hence  by  the  preceding  theorem, 
(i-d)  also  implies  (i-c). 

The  concept  of  xiniform  stability  for  deterministic  systems  has  its 
natural  counterpart  in  stochastic  systems. 

DEFINITIONS  (ii-a)-(U-f):  The  trivial  solution  if  said  to  be  uni¬ 
formly  stable  mean  norm,  j^t]^  m^an  square,  ^  ^  moment. 
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in  probability,  >lmo»t  •urely,  almoat  uniformly -in-w,  if  it  satisfies 
definitions  (i-a),  (i-b),  (i-c),  (i-d),  (i-e),  (i-f)  respectively,  with  6 
independent  of  t^  (and  B  independent  of  tQ  in  definition  (i-f))* 

The  following  theorem  parallels  Theorem  1: 

THEOREM  2:  The  trivial  solution  of  (2.  2)  is  uniformly  stable  al¬ 
most  surely  if  and  only  if  it  is  uniformly  stable  almost  uniformly -in-d). 

PROOF:  The  proof  follows  at  once  from  the  proof  of  Theorem  1 
upon  noting  the  uniformity  in  t^  of  B  and  of  t. 

ASYMPTOTIC  CONSIDERATIONS.  In  the  preceding  section,  the 
definitions  of  stability  relate  to  the  behavior  of  the  solution  function 
for  aU  time  t  >  t^  .  In  this  section  it  is  the  large  time  or  asymptotic 
behavior  of  the  solution  functions  that  is  of  interest.  Additional  re¬ 
quirements  of  appropriate  behavior  for  all  time  t  >  t^  may  or  may  not 
be  imposed. 

The  collection  of  solution  functions  g(t,  t^;  u )  is  a  class,  indexed 

by  (xq,  tg),  of  random  processes.  The  convergence  in  t  of  the  random 

processes  for  all  (x^ft^)  belonging  to  some  set  is  of  interest.  The  types 

of  convergence  considered  are  those  commonly  studied  in  probability 

theory,  namely,  in  the  ith  mean,  in  probability,  almost  sure,  and  al- 

21 

moat  uniform  (-in-w).  (See  Loeve.  )  From  the  usual  definitions  of 
these  modes  of  convergence  the  definitions  of  stability  follow,  attention 
being  paid  to  the  dependence  of  the  random  processes  on  the  couple 


DEFINITIONS:  The  trivial  solution  is  called 
(iii-a)  quasi- asymptotically  stable  in  the  meam  norm  if  given  t^  there 
exists  6(t^  >  0  such  that  ||  x^  |j  <6  implies  E  ||  g(t,  x^,  t^)  ||  — >  0  as 
t  — ^oo. 

(iii-b)  <^uasi-asymptotically  stable  ii^t^  mean  square  if  given  tQ  there 

exists  i(tQ)  >  0  such  that  U  Xq|]  <  i  implies 
o  u 

E  ^  t-hoo. 

j=l 
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qwi.«l->«ymptotic>lly  itable  In  the  itk  moment  if  giy«m  tkare 
exists  6(tg)  >  0  suck  that  |  XpH  <  6  implies  Egj  jj(t,Xp,tp)  as 
t  — ^co. 

(iii-d)  q^nasi-asymptotically  stable  in  probability  if  given  r\  >  0>  tp, 
there  exists  6(q,tp)  >  0  such  that  given  any  «  >  0  there  is  a 
T(€  ,  n,  Xp,  tp)  >  0  such  that  |  Xp  [j  <6  implies 
P[ll  g(t.Xo*Vl^  ]  >1  '  n  for  all  t  >tp  +  T. 

(iii-e)  quasi- asymptotically  stable  almost  surely  if  there  exists  an  w-set 
.S  with  P(S)  s  1  such  that  for  every  usS  the  trivial  solution  is  quasi- 
asymptotically  stable;  or  equivalently,  there  exists  an  u- set  S  with 
P(S)  s  1  such  that  given  tp,  w  c  S,  there  exists  6(w ,  tp)  >  0,  azid  for  any 
t  >  0,  there  is  a  T(  c  ,(i>,Xp,  tp)  >  0  such  that  ||  Xp  j|  <  fi  implies 
I  g(t,  Xp,  tp!  w)  II  <  €  for  all  t  >  tp  +  T. 

(iii-f)  quasi-asymptottcally  stable  almost  oniforoa^-in-u  if  given  q  >  0,tp, 
there  exists  6(^1,  tp)  >  0,  and  given  any  Xp,  ||  Xp||  <  d,  there  is  an  w-set 
V  P(B)  >1-11  such  that  ||  Xp|  <  i  implies  g(t,  Xp,  tp{  u)— bO 
as  t— >00  uniformly  on  B;  or  equivalently,  given  q  >  0,  tp,  there  exists 
fg)  >  0  such  that  I  Xp||  <6  isaplies  there  exists  an  w-set  B(il,Xp, tp) 
with  P(B)  >  1-1)  having  the  property  that:  to  every  c  >  0,  there  corre¬ 
sponds  a  T(€  ,  q,  Xp,  tp)  >  6  such  that  |  Xp  ||  <  *  implies  ||  g(t,  Xp,  tp!  w)  || 

<  c  for  all  t  >  tp  +  T  and  all  w  c  B. 

The  discussion  given  at  the  end  of  definition  (ii-e)  is  equally  per¬ 
tinent  here.  In  definition  (iii-e)  the  bound  on  the  initial  value  and  the 
number  T  governing  the  rate  of  convergence  depend  on  w;  in  definition 
(iii-f)  both  6  and  T  are  uniform  in  w'  but  this  uniformity  is  over  an 
w  -set  B  whose  probability  can  be  chosen  to  be  arbitrarily  close  to, 
but  not  always  equal  to,  one.  The  following  two  theorems  complement 
Theorem  I. 

THEOREM  3:  The  trivial  solution  of  (2.  2)  is  quasi-asymptoti- 
cally  stable  almost  surely  if  it  is  quasi-asymptotically  stable  almost 
uniformly  -  in- w  with  the  set  B(see  definition  (iii-f)  above)  indepen¬ 

dent  of  Xp.  (Proof:  See  Appendix  C.  ) 
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THEOREM  4:  trivial  solution  of  (2.  2)  is  quasi-asymptotically 

stable  almost  uniformly- in- w if  it  is  quasi-asymptotically  stable  almost 
surely.  (Proof:  See  Appendix  C.  ) 

Clearly  (lli-f)  implies  (iii-c),  whence  by  Theorem  4  (iii-d)  also 
Implies  (iii-c). 

The  statements  of  the  above  theorems  can  be  suitably  altered  to 
incorporate  some  requirement  of  uniformity  of  fi,  B,  T,  in  the  variables 
Xp,  tp.  The  proofs  would  parallel  closely  those  of  Theorems  1-4.  Defi¬ 
nitions  (i-d)-(i-f),  (ii-d)-(ii-f),  (iii-d)- (iii-f)  can  also  be  modified  some¬ 
what  to  describe  closely  related  concepts.  For  example,  definitions 
(i-e),  (ili-e)  may  be  altered  so  that  the  set  S  of  unit  probability  depends 
on  the  choice  of  tp.  Thus  the  deHnitions  and  theorems  given  in  this  sec¬ 
tion  are  merely  indicative  of  the  concepts  and  results  that  can  be  ob¬ 
tained.  They  are  by  no  means  exhaustive. 

In  the  deterministic  case,  asymptotic  stability  requires  both  stabili¬ 
ty  and  quasi-asymptotic  stability.  Likewise,  in  the  random  case,  the 
following  definitions  can  be  formulated: 

DEFINITION:  The  trivial  solution  is  called 

(iv-a)  asymptotically  stable  in  the  mean  norm  if  it  is  (ii-a)  and 
(iii-a). 

(iv-b)  asymptotically  stable  in  the  mean  square  if  it  is  (ii-b) 
and  (iU-b). 

(iv-c)  asymptotically  stable  in  the  ith  moment  if  it  is  (ii-c)  and 
(iii-c). 

(iv-d)  asym ptotically  stable  in  probability  if  it  is  (ii-d)  and  (iii-d). 

(iv-e)  asymptotically  stable  almost  surely  if  it  is  (ii-e)  and  (iii-e). 

(iv-f)  asymptotically  stable  almost  uniformly -in-  m  if  it  is  (ii-f) 
and  (iii-f). 

It  follows  from  their  respective  definitions  that  (iv-f)  implies 
(iv-d);  hence  by  the  above  theorem,  (iv-e)  also  implies  (iv-b). 
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The  remaining  concepts  of  stability,  (v)  -  (ix),  can  be  similarly 
generalized  to  the  random  case.  To  avoid  repetitiousness,  they  will 
not  be  discussed  here.  Exponential  asymptotic  stability  is,  however, 
considered  briefly  for  it  is  used  specifically  in  Chap.  V. 

DEFINITION:  The  trivial  solution  is  called 

(ix-a)  exponentially  asymptotically  stable  in  the  mean  norm  if 
there  exists  a  v  >  0  ,  and  given  any  c  >  0  there  is  a  6(c  )  >  0  such 
that  for  all  tg,  II  Xq II  <e  implies  E  ||  g(t,XQ,tp)||  <€  exp [ -v(t-tQ)] 
for  all  t_>  tp. 

(ix-c)  exponentially  asymptotically  stable  in  the  ith  moment  if  there 
exists  a  V  >  0,  and  given  any  c  >  0  there  is  a  6(c  )  >  0  such  that  for  all 
‘o*  i  *oll  ^  ^  implies  ||  Egj  jj(t  ,Xq  ,tQ)  ||  <e  exp]  -  v(t-tQ)]  for  all  t>  t^. 

As  before  ,  (ix-c)  for  even  i  implies  (ix-a).  If  the  system  is  linear, 
that  is  ,  if  f  in  Eq.  (2.  2)  is  almost  surely  linear  in  x;,  then  (ix-a)  implies 
uniform  asymptotic  stability  in  the  mean  norm  and  (ix-c)  implies  uniform 
asymptotic  stability  in  the  ith  moment.  Moreover ,  by  virtue  of  the 
linearity  in  x  of  the  solution  functions  of  the  differential  system,  for 
both  (ix-a)  and  ix-c)  there  exists  a  constant  K  >  1  such  that  it  is  possible 
to  choose  6(c)  =  c  /K  so  that  E  1|  g(t,XQ,tQ)||  <  K||  Xq||  exp  [-v(t-tQ)]  and 
|Egj  .j(t,XQ,tQ)||  <K||xQ||exp[  -  v(t-tQ)]  respectively. 

m.  LINEAR  SYSTEMS  WITH  INDEPENDENT  PARAMETERS 
SUMMARY 

This  section  is  devoted  to  the  stability  analysis  of  the  random 
linear  differential  system  with  piecewise  constant  coefficients; 

X  =  *  ^k-1  —  ^  ^  ’  k=12...  (i3. 1) 

where  x(t)  is  an  n-vector  and  A^  is  a  constant  n  x  n  matrix.  The  be¬ 
havior  of  the  system  is  completely  determined  by  the  random  sequence 
{A^(t^-t^  ^)}  here  assumed  to  be  an  independent,  identically  distri- 


buted',  random  process.  Chapter  IV  treats  the  case  where 


-17- 


is  a  homogeneous  Markov  chain.  *  Necessary  and  sufficient  conditions 
for  stability,  and  (exponential)  asymptotic  stability,  in  the  ith  moment, 
are  obtained.  It  is  shown  that  almost  sure  asymptotic  stability  is  a 
consequence  of  asymptotic  stability  in  the  ith  moment,  i  an  even  integer, 
and  that  the  converse  implication  is  false. 


3.1  ith  Moment  Stability 

Given  an  initial  vector  xCt^)  =  the  solution  of  the  differential 
equation  (3. 1)  in  the  first  time  interval  is 


x(t)  =  e  , 

Defining  the  solution  function  at  time  t^  by  continuity  from  the  left, 

M\)  =  e  Xq  , 

and  taking  this  to  be  the  initial  value  for  the  next  time  interval,  the 
solution  in  the  second  time  interval  is 


x(t)  =  e 


x(tj) 


s  e 


^^t-tj)  Aj(t^-tQ) 


c  *0  ’ 

It  follows  that  the  time  function 


x(t)  =  e 


Sc-1< 


t  <  t. 


^(ti.to) 

.  e  Xp  , 

k=  1,2,3,... 


(3.2) 


satisfies  the  initial  condition  x(tQ)  =  x^,  ajid  satisfies  the  differential 
equation  (3. 1)  everywhere  except  at  the  time  instants  t^,  t^,  t^,  .... 
The  function  x(t)  defined  by  (3.  2)  will  be  taken  as  the  unique  solution 
of  (3. 1)  satisfying  the  initial  condition  x(tQ)  =  x^. 

Define 

=  e  ^  (3  3) 

All  of  the  results  of  this  chapter  can  be  deduced  from  those  of 
ChitplV  (and  Appendix  E)  which  treats  the  Markov  case.  It  is  prefer¬ 
able,  however,  from  an  expository  point  of  view,  to  treat  first  the 
simpler  case  of  independence. 
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THen  the  solution  (3.  2)  caji  be  written  as 


A.(t-t.  i) 

*(t)  =  e  *  5k-l^k-2’  ••^1*0 

tj^_j  ^  ^  ^  h  =  1,  2,  ... 

Eralxiating  the  solution  x(t)  at  times  t^  only,  (3.  4)  yields 


*k  =  «  kJk-r  •  •  Vo 

where 

=  *k  =  ^*k-l 


(3.4) 


(3.  5) 

(3.6) 


It  is  clear  from  the  definition  of  £q.  (3.  3),  that  the  behavior  of 
depends  mly  on  the  probabilistic  structure  of  the  random  sequence 
W<V*k.  ^)}.  It  is  also  clear  from  the  continuity,  with  respect  to 
the  initial  value  x^,  of  solutions  of  the  differential  equation  (3.1)  that 
it  suffices  to  examine  the  sequence  {x^^}  to  determine  the  stability 
(of  the  trivial  solution)  of  the  differential  system  (3. 1),  provided  the 
^  are  bounded  in  some  sense.  Hence  it  will  be  assumed  that  the 
are  bounded  (almost  surely)  \miformly  in  k,  i.  e. ,  there  exists 
a  finite  M  such  that  for  all  k,  Pf  |  <  M]  =  1. 

In  the  remainder  of  this  section  it  is  assumed  that  {  is  a  se¬ 
quence  of  independent,  identically  distributed,  random  noatrices  of 
order  n  x  n.  Since  the  sequence  {x^^}  is  generated  by  subjecting  a 
given  vector  x^  to  a  succession  of  independent,  identically  distributed, 
random  linear  transformations:  x^^  =  * '  ^1^0*  random 

sequence  (x^^)  is  a  homogeneous,  nonstationary,  infinite  Markov 
chain.  (See  Chap.  IV,  Secs.  4. 1  and  4.  2.  ) 

Take  the  expectation  of  both  sides  of  £q.  (3.  5)  to  obtain 


EXk  =  ^^5k^k-r  •  •  J  1^*0’ 

which  by  independence  becomes 

EXj^  =  (E|j^)(Eij^_j)...(E|j)xQ 
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I 


Moreover,  lince  the  have  a  common  distribution, 


(3.  7) 


where  the  subscript  on  S  has  been  dropped  because  E  S,  does 

j 

not  depend  on  j.  , 

Equation  (3.  7)  shows  that  the  behavior  of  Ex^  as  a  function  of 
k  depends  only  on  E{  (for  fixed  x^),  in  particular,  only  on  the 
location  and  the  index  of  the  eigenvalues  of  E^  .  This  point  is 
elaborated  on  later  when  it  is  shown  that  the  computation  of  the  ith 

moment,  i  ^  1,  2,  3 . ^  leads  to  equations  identical  in  form 

to  (3.  7). 

To  evaluate  the  second  moment  it  is  necessary  to  work  with 
the  squares  of  the  components  of  x^.  For  algebraic  simplicity  assume 
that  the  system  is  order  (n=)  2.  Let  x^^  =  (vi^,  v^).  Then  the  relation 
Xj^  =  J  written  as 


[3'[ 


r-k-n 

4>-l  Lv,J 


which  may  be  expressed  as 


u  + 

ni  '^k-l  ^ 

^k  = 

^21  '^k-l  ^ 

'"k-l 

"^k-l 


-20- 


Form  all  lecond-order  products  of  components  of  to  obtain 


2  2  ^x(lt)x(k)  «  xaWJIk)  x^(k)2  2 

“k-1  ■*■^11  ^12  "k-l^k-l'*' 4i2  ‘•il  ^k-l^-l‘*''^12  '^k-l 

“k^k"  ^11  ^21  'Ht-r  11  ^22  '^-l'^k-rn2  ^21  ^k-l^-r  ^2  T22^k-1 
'^k^  ^21  ^11  Tc-r  21  -^12  “k-rk-r  ^22  ^11  V^k-r  ^2  ^12  ^k-1 


(3.8) 


Define 


‘k(2] 


by 


*k[2]  = 


2 

“k 


“k'^k 


Then  (3,  7)  shows  that  ^^2]  a  linear  transform  of  l[2]' 
*k[2]  =  ^k[2]*k-l[2]’ 


the  matrix  ^  transformation  being  defined  by  the  set  of 

equations  (3.  8).  The  recursion  equation  (3.  9)  is  linear,  as  is  the 
original  equation  (3.  6),  yet  it  relates  second'Order  forms  of  compo¬ 
nents  of  Xj^,  this  being  achieved  at  the  cost  of  working  in  a  higher 
dimension.  The  technique  used  is,  in  fact,  that  of  Kronecker  pro¬ 
ducts  (as  explained  below),  which  dates  back  to  the  last  century 

22  14 

(see  MacDuffee,  pp.  81-86)  and  which  was  used  recently  by  Bellmsji 

in  solving  a  similar  problem  of  computing  moments. 
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The  matrix  ^2]  ^ 

Kronecker  (or  direct)  .product  of 


(3.  9)  is  recognized  to  be  the 
by  itself: 


likewise,  *55^2] 


is  the  Kronecker  product  of  by  itself: 


*k[2]  '  S 


(Appendix  A  is  a  brief  outline  of  some  of  the  useful  properties  of 
Kronecker  and  power  products  of  matrices.  ) 

By  using  the  theory  of  Kronecker  products,  Eq.  (3.  6),  ralid  for 
systems  of  any  order,  can  be  easily  derived.  Take  the  self- Kronecker 
product  on  both  sides  of  Eq.  (3.  6)  to  obtain 


*k  ®  ■Si '  <*k*k>  ®  <’•“> 

where  x^  is  an  n-vector.  Now,  for  any  two  matrices  A  and  B,  not 
necessarily  square  but  such  that  the  product  AB  is  defined. 


AB  ®  AB  =  (AB)^2]  =  '^[2]  ^[2]  =  ®  ® 

that  is,  the  operations  of  the  Kronecker  product  and  the  conventional 
matrix  product  commute.  It  is  this  commutative  property  that  makes 
the  theory  of  Kronecker  products  so  applicable  to  the  problem  of 
evaluating  moments.  For  it  follows  from  Eq.  (3. 11)  that  (3. 10)  can 
be  reduced  to 

*k[2]  =  ^k[2]*k-l[2]  ’ 

whence  by  iteration  on  k 

*k[2]  =  ^k[2]^k-l[2]  •  •  •  ^1[2]*0[2] 


.22- 


Similarly,  for  ith-order  products,  form  i-fold  products  from  Eq.  (3.6), 

»  Vk.i  ® . . .  ® 

and  by  repeatedly  using  the  commutative  property  (3.11),  obtain 


*k[i]  =  ^k[i]^k-l[i]  '-  ^l[l]*0[i]  • 

where  the  subscript  [i]  denotes  the  ith  self- Krone  eke  r  product. 

Now,  {  is  an  independent,  identically  distributed,  random 

sequence  of  matrices  (of  order  n^).  Hence,  taking  the  expectation  in 
Eq.  (3.12), 


^*k[i]  =(^^[i])%[i] 


(3.13) 


The  vector  ii^  components  all  ith-order  products  of  com¬ 

ponents  x^;  hence,  Eq.  (3.13)  gives  an  explicit  and  simple  relationship 
for  studying  the  ith  moment  of  components  of  x^.  The  knowledge  of 

the  moments  at  time  t.  may  be  used  to  study  the  probability  distribution « 

22*^ 

of  (see  Cramer,  p.  176).  Also,  bounds  on  the  probability  of  devi¬ 
ation  from  zero  of  any  component  of  x,  may  be  evaluated  readily  by 
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using  the  Markov  inequality 


P[|  Z|  >«  ]  < 


E  Z 


e  >0, 


(3. 14) 


where  Z  is  a  scalar  random  variable  and  r  is  a  positive  real  number. 

Equation  (3. 13)  shows  that  the  behavior  of  large  powers  of  a 
matrix  is  of  interest.  Hence  consider  the  following 

LEMMA:  Let  F  be  a  complex  matrix  of  finite  order.  Then 

(i)  lim  =  0  if  and  only  if  every  eigenvalue  of  F  lies  inside 
k-+o>  , 

the  unit  circle;  in  fact,  will  converge  exponentially: 

there  exist  two  constants  K,  >  1,  >  0  such  that 

|P^|  <  exp[  ; 
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(ii)  lim  F  s  ^  0  if  and  only  if  one  is  an  eigenvalue  of  F  of 
k-^oo 

>(< 

index  one  and  all  other  eigenvalues  of  F  are  inside  the  unit  circle; 

(iii)  F  is  bounded  uniformly  in  k  but  does  not  converge  as 
k— >00  (is  "oscillatory")  if  and  only  if  F  has  no  eigenvalues  outside 
the  unit  circle,  all  eigenvalues  on  the  unit  circle  are  of  index  one, 
and  there  exists  an  eigenvalue  not  equal  to  one  on  the  unit  circle; 

(iv)  lim  II  F^ll  =  00  if  and  only  if  F  has  an  eigenvalue  outside  the 
k-^oo 

unit  circle  and/or  has  eigenvalues  of  index  greater  than  one  on  the 
unit  circle^  in  fact,  ||  F^||  will  diverge  exponentially:  there  exist 
two  constants  1  ,  y^>  0  such  that  ||  F^||  > K^xp[+ij^k]. 

PROOF:  To  arrive  at  the  above  conclusions  ,  it  is  only  necessary 
to  transform  F  into  a  Jordan  canonical  form  so  that  powers  of  F  can 
be  studied  readily. 


THEOREM  1:  The  trivial  solution  is  stable  in  the  ith  moment  (A) 
if  and  (A')  only  if  no  eigenvalue  of  E§  lisB  outside  the  unit  circle 
and  any  eigenvalue  on  the  unit  circle  has  index  one.  It  is  asymptotically 
stable  in  the  ith  moment  (B)  if  and  (B')  only  if  all  eigenvalues  of 
lie  inside  the  unit  circle.  Moreover,  in  (A)  and  (B)  it  is  uniformly 
stable  in  the  ith  moment  and  uniformly  asymptotically  stable  in  the 
ith  moment.respectively. 

PROOF:  (A)  By  the  assumptions  on  the  eigenvalues  and  by  the 

above  Lemma,  sup 
k>l 


<  M  for  some  finite  M  Given  any 


(3.15) 


€  >  0,  choose  6.(()  =  c/M.  Then  by  Eq.  (3.13), 

l|ExK[i]ll  llxoCijII  ■ 

“  II  *o[  i]  II  ^i’  II  ^*k[  i]  II  II  *0[  i] 

is  equivalent  to  ||  Xq||  <  6  for  some  6  >  0  .  Furthermore,  6',  and 
hence  6,  do  not  depend  on  t^.  Hence  the  trivial  solution  is  uniformly 


<  6. 
1 


stable  in  the  ith  moment. 


(A')  If  the  eigenvalue  assumption  is  violated,  then  by  part  (iv) 

of  the  above  Lemma,  ||(E  {r  .i)^|| — ^  oo  as  k-- >oo;  or  equivalently, 

*  k 

some  element  of  the  matrix  (E2|’  )  becomes  unbounded  as  k — >oo. 


The  index  of  an  eigenvalue  is  its  multiplicity  as  a  root  of  the 
minimal  polynomial  of  F.  See  also  Friedman,  Chap.  2. 
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Hence  there  does  not  exist  stny  6  >  0  witii  the  property  that  for  some 

‘o’  i]  i]  II  remains  botmded  as  k->oo  for 

all  II  XqII  <  £.  This  implies  that  the  trivial  solution  cannot  be  stable 
in  the  ith  moment. 

(B)  By  part  (i)  of  the  preceding  Lemma,(|:{^  ^j)  .-^0  as  kHK». 

Clearly  this  convergence  is  uniform  in  t^.  Hence,  given  any  6  >  0, 
it  follows  by  £q.  (3. 15)  that  for  all  t^,  ||  ^qII  ^  ^  implies  ^0 

as  k — >00  uniformly  in  t^,  and  uniformly  on  |  Xq|  <6  (since  the 
system  is  linear).  Hence  the  trivial  solution  is  uniformly  quasi-asymp- 
totically  stable  in  the  ith  moment.  Moreover  by  (A),  it  is  uniformly 
stable  in  the  ith  moment.  Therefore  it  is  uniformly  asymptotically 
stable  in  the  ith  moment. 

(BV)  If  every  eigenvalue  of  inside  the  unit  circle,  then 

by  part  (i)  of  the  preceding  Lemma,  will  not  converge  to  the 

zero  matrix  as  k— ^oo.  Hence  there  exists  no  6  >  0  such  that  for  some 

*0' ®*k[i]  "  ^‘»>^^.J*oll  ^ 

fore  the  trivial  solution  cannot  be  quasi>asymptotically  stable  in  the  ith 
moment  and  hence  asymptotically  stable  in  the  ith  moment. 

For  subsequent  use.esptBciallj  in  Chap.  V  in  connection  with  the  in¬ 
homogeneous  (forced)  random  linear  system,  it  is  useful  to  establish 
the  property  of  exponential  asymptotic  stability: 

THEOREM  2:  For  the  random  linear  system  (3.1),  the  following 
propositions  are  equivalent: 

(a)  All  eigenvalues  of  E^  inside  the  unit  circle. 

(b)  The  trivial  solution  is  uniformly  asymptotically  stable  in  the 
ith  moment. 

(c)  The  trivial  solution  is  exponentially  asymptotically  stable  in 
the  ith  moment. 

PROOF:  The  equivalence  of  (a)  and  (b)  was  established  in  Theorem  1. 

That  (c)  implies  (b)  follows  at  once  from  the  definitions  of  exponen¬ 
tial  and  uniform  asymptotic  stability  in  the  ith  moment. 
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To  show  that  (a)  implies  (c):  Since  all  eigenvalues  of  ^ j 
are  inside  the  unit  circle,  by  part  (iv)  of  the  Lemma  there 


are  two  constants  K  >  1  and  v  >  0  such  that 
Hence  by  Eq.  (3.13), 


ki 


<  K  exp(-vk). 


II  ^*k[i]  II  ^  ^11  *0[i]  II  <  kJI  XqH  exp(-vk) 


for  some  K.  >  1  from  which  follows  the  desired  result. 

An  inspection  of  the  set  of  equations  (3.  8)  reveals  that  the  third 
equation  is  superfluous  and  leads  to  the  condensed  set  of  equations 


2' 

'^k 

i(k)2 

11 

2<|>(k)<))(k) 

11  12 

((.(k)2  “1 

12 

1 

c 

_ L 

'^k'^k 

= 

4,^4,  (k) 

Ml  ^21 

4,^4,  (k) 
^12  ^22 

"k-l'^k-l 

2 

L  '^kJ 

fcl«2l 

4>W2 

^22 

2 

- 

which  may  be  expressed  as 


*k(2)  *  ^k(2)*k-l(2) 


The  vector  matrix  ^^2)  recognized  to  be  the  self¬ 
power  products  (see  Appendix  A)  of  Xj^  and  respectively.  As  in 

the  Kronecker  product  case,  for  a  system  of  any  order, 

^*k(i)  =  <^5(i)^*'*0(i) 

is  the  relation  for  the  ith  moment  and  the  eigenvalues  of  deter¬ 

mine  the  asymptotic  behavior  of  the  ith  moment. 

In  theoretical  analysis,  the  symmetrical  structure  of  the  Kronecker 
product  matrix  is  most  useful;  on  the  other  hand,  the  smaller  order  of 
the  corresponding  power  product  matrix  can  be  distinctly  advantageous 
in  reducing  the  numerical  work.  The  Kronecker  product  will  be  used 
exclusively  hereafter. 
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A  central  problem  in  the  application  of  the  above  methods  is  the 

study  of  the  location  of  the  eigenvalues  of  matrices  with  respect  to  the 

unit  circle.  Either  Schur-Cohn  or  transformed  Routh-Hurwitz  criteria 
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may  be  used  (see  Marden,  Bharucha,  Jury  '  )  but  paper -and- 

pencil  methods  are  tedious  even  for  systems  of  low  order.  The  prob¬ 
lem,  however,  readily  lends  itself  to  solution  on  a  digital  computer  . 


3  .2  Almost  Sure  Asymptotic  Stability 

By  Eq.  (3.13)  and  Theorem  2,  if  all  the  eigenvalues  of  E  are 

inside  the  unit  circle,  then  the  trivial  solution  is  asymptotically  stable 

in  the  ith  moment  and  Ex,  r  converges  exponentially  to  zero.  The 

oo 


exponential  convergence  implies  that 


1 

k=l 


exists  and  is  finite. 


which  via  the  Markov  inequality  and  the  Borel-Cantelli  lemma  shows 
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that  x^^— ‘  ■  >  0  if  i  is  even.  Since  the  system  is  linear,  this  conver¬ 
gence  is  uniform  in  Xq,  from  which  follows 


THEOREM  3:  The  trivial  solution  is  equiasymptotically  stable 
almost  surely  if  for  some  even  i,  all  eigenvalues  of  inside 


the  unit  circle 


PROOF:  By  Eq  .  (3  .13) 


Summing  on  k, 
cx> 

1  = 
k=l 


CD 


1 

k=l 


(3.16) 


since  all  eigenvalues  of  moduli  less  than  one. 

Let  u^  be  any  component  of  x^^.  Then  Euj^  is  a  component  of 

oo 

EXkji],  and  by  (3.15),  ^  Euj^  <  oo  .  But  by  the  Markov  inequality, 

k=l 
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(3.14), 


E  u. 


p[KI  ^  J  — r 


‘i>o- 


Hence  ^P[  |  Uj^|  >  «j]  <  “>  every  € ^  >  0,  which  by  the  Borel-Cantelli 
lemma  implies  that  %  0. 

Therefore,  given  any  6  >  0,  for  all  tp,  ||  Xq||  <6  implies 
g(t,  Xq,  tQ)— ^0.  Moreover,  since  the  system  is  linear  the  convergence 
to  zero  is  uniform  on  ||  x^jl  <  6.  Hence  the  trivial  solution  is  quasi- 
equiasymptotically  stable  almost  surely 

^0 


given  any  €  >  0,  t^,  there  exists  a  T(6  ,  6,  >0  such  that  „  -.q. 


Also,  for  almost  every  u, 

*0 II 

implies  ||  g(t,  x^,  t^;  w)||  <€  for  all  t  >  t^  +  T,  Now  T  is  independent 

of  Xq,  II  xqII  <  6,  and  g(t,  x^,  tQ;u)  is  continuous  in  x^;  hence 
max  II  g(t,  X-,  t.;  w)  II  is  continuous  in  Xq.  Therefore  there 


exists  a  6’(«  .t^)  >0,  6'  <  6,  such  that  ||  Xq||  <  6’  implies 
II  g(t,  Xq,  tQ;« )  II  <  €  for  all  t  >  t^  +T  (and  also  for  t^  <  t  <  t^  +  T 
by  definition  of  6'  and  T).  Consequently  the  trivial  solution  is  stable 
almost  surely.  (This  also  could  have  been  derived  by  using  the  linear¬ 
ity  in  Xq,  instead  of  using  the  continuity  in  x^,  of  the  solution  function.  ) 
Since  the  trivial  solution  is,  in  addition,  quasi- equiasymptotically  sta¬ 
ble  almost  surely,  it  is  equiasymptotically  stable  almost  surely.  Note 
that  the  only  reason  the  solution  is  not  uniformly  asymptotically  stable 
almost  surely  is  that  T  depends  on  t^,  that  is,  the  convergence  to  zero 
is  not  necessarily  uniform  in  t^. 

The  converse  of  Theorem  3  is  not  true:  Example  1  below  shows 
that  the  trivial  solution  can  be  almost  surely  quasi-asymptotically 
stable  although  for  all  i,  eigenvalues  outside  the  unit  circle. 

This  is  in  contrast  to  Theorem  1  which  states  that  asymptotic  stability 
in  the  ith  moment  is  equivalent  to  having  all  eigenvalues  of 
inside  the  unit  circle. 
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3.3  Example  1 


Let  =  T  (fixed) 


bj  <  0  with  probability  p 


=  ^2  —  ®  with  probability  1-p 


Then 


b,T 


J  =  e  ^  with  probability  p 


b^T 


=  e  with  probability  1-p 


and 


ib,T 


1 

=  e  with  probability  p 


ib2T 


^k[i]  ”  *  with  probability  1-p. 


ib  T 

1  +  (l-p)e  By  Theorem  1,  the  trivial  solu- 


Hence  =  P® 

tion  is  stable  in  the  ith  moment  if  and  only  if 


e  -  1 

e  -  e 

It  is  asymptotically  stable  in  the  ith  moment  if  the  above  constraint  is 

changed  to  a  strict  inequality.  These  results  are  exhibited  graphically 

in  Fig.  1.  Note  that  the  region  of  stability  increases  as  b. ,  b,  decrease. 

-1-1  1  ^ 

This  is  to  be  expected  for  b^  ,  are  the  respective  "time  constants" 
of  the  stable  and  unstable  modes  of  the  system. 
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It  is  interesting  to  examine  the  situation  when  bj  =  -oo^  =  e  =  0, 

T  ^  0,  with  probability  p.  The  curve  in  Fig.  1  delineating  the  region 

of  stability  now  starts  at  the  origin  and  rises  exponentially  to  one.  But 

for  all  p  >  0,  T  >  0,  the  trivial  solution  is  a.  s,  asymptotically  stable 

because  the  only  realization  that  does  not  become  zero  as  t — ^oo  is 

biT 

the  one  where  the  "stable"  mode  e  is  never  achieved  and  thus  has 
probability  (l-p)°°  =  0.  However,  it  should  be  noted  that  for  any  p  <  1, 
there  exists  a  positive  probability  that  the  trivial  solution  is  unboxmded^' 
for,  given  any  bound  M  >  0,  ||  x(t)||  will  exceed  M  if  the  system  starts 
in  and  remains  in  the  unstable  mode  for  a  finite  length  of  time. 

3.4  Example  2 

The  time  intervals  fixed  length  T.  The  system  has 

a  double  "pole"  at  a  <  0  and  b  >  0  with  probabilities  p  and  1-p,  respec- 


with  probability  p 

with  probability  1-p 
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•.T  bT 

The  corresponding  values  of  are  e  B  and  e  where 

•■c :] 


Figure  2 


For  example,  the  feedback  control  system  shown  above  would  have 
a  =  -1,  b  =  1  if  the  random  gain  K  was  equal  to  4  or  0  with  respective 
probabilities  p  and  1-p. 

Returning  to  the  general  case, 

El  =  (pe^*^  +(l-p)e^’^}B 

B  has  a  double  eigenvalue  at  +1.  Hence  the  eigenvalues  of  are 
inside  the  unit  circle  if  and  only  if 

pe*^  +  (l>p)e^^  <  1, 
that  is,  if  and  only  if 


e  -  e 


in  which  case  Ex  =  0. 

00 

.  Again, 

E5[2J  = 
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Note  that  +1  ia  an  eigenvalue  of  B  of  multiplicity  two.  Hence,  +1 
is  an  eigenvalue  of  B  ^  2j  multiplicity  four  (see  Appendix  A,  Eq.  A.  5). 
Moreover,  it  can  be  shown  that  its  index  is  also  four. 

The  eigenvalues  of  ^^2]  inside  the  unit  circle  if  and  only  if 

2bT 

(3.17) 


.  e  -1 

“ZBT 


-e 


7ST 


in  which  case  ^  k-  ^oo  and  the  system  is  asymptotically 

stable  in  the  second  moment.  If  (3. 17)  is  changed  to  an  equality,  the 
system  becomes  unstable  in  the  second  moment  for  ^$[2]  now  has  one 
as  an  eigenvalue  of  index  four  on  the  unit  circle.  Figure  3  summarizes 
the  above  results. 


Asymptotically  stable  in  the  second  moment;  [l] 

Unstable  in  the  second  moment:  [2]  -  [5] 

Ultimate  mean  zero:  (ll  -  [3] 

Ultimate  mean  infinite;  [4]  -  [5] 

Figure  3 

Suppose  a  =  -1  and  b  =  0.  Then  this  example  reduces  to  Example 

3 

1  of  Bertram  and  Sarachik  where  by  Lyapunov's  second  method,  the 

-T 

authors  arrive  at  the  result  that  p  >  l/(l-e  )  is  a  sufficient  condition 
for  asymptotic  stability  in  the  mean.  But  Eq.  (3. 17)  implies  that  if 
a  <  0,  b  =  0,  a  necessary  and  sufficient  condition  for  asymptotic  sta¬ 
bility  in  the  mean  is  p  >  0.  Indeed,  by  using  the  results  of  the  next 
section,  it  can  be  shown  that  the  latter  result  is  true  even  if  the  se¬ 
quence  is  any  finite-order  MarkbV  chain  with  a  nonzero  transi¬ 

tion  probability  of  going  from  the  unstable  to  the  stable  mode. 
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IV.  LINEAR  SYSTEM  WITH  MARKOV  PARAMETERS 


Summary 

The  problem  considered  here  is  identical  to  the  one  treated  in 
Chap.III  except  that  the  succession  of  modes  of  operation  of  the  random 
linear  system  forms  a  Markov  chain  rather  than  just  an  independent, 
identically  distributed,  random  sequence.  The  differential  equation, 
as  before,  is 

X  =  ^k-1  —  ^  ^  ^k*  h  =  1,  2,  ... 

where  x(t)  is  an  n-vector  and  A^  is  a  constant  n  x  n  matrix.  The  system 
behavior  is  completely  described  by  the  random  sequence 

tj^-t^  p} ,  here  assumed  to  be  a  Markov  chain.  Sufficient  conditions 
for  stability,  and  (exponential)  asymptotic  stability,  in  the  ith  moment, 
are  obtained.  It  is  shown  that  these  conditions  without  further  qualifi¬ 
cations  are  not  also  necessary.  The  dependence  of  the  ith  moment  of 
the  solution  function  on  the  initial  distribution  and  on  the  types  of  modes 
(states)  of  the  Markov  chain  is  discussed.  It  is  shown  that  the  sufficient 
conditions  for  asymptotic  stability  in  the  ith  moment,  i  an  even  integer, 
also  imply  almost  sure  asymptotic  stability. 

4.  1  Markov  Chains 

This  section  outlines  some  of  the  pertinent  facts  of  the  theory  of 

Markov  chains.  Unfortunately  there  exists  no  stamdard  terminology  in 

the  literature  of  Markov  chains;  a  major  purpose  of  this  section  is  to 

introduce  the  notation  and  terminology  to  be  used  later  in  the  section. 

29 

Some  general  references  on  Markov  chains  are  Bharucha-Reid, 

30  31  32 

Doob,  Gantmacher,  and  Kemeny  and  Snell. 

A  Markov  process  is  an  indexed  set  {Z^,  ttTo(-oo,  oo)} 

of  random  variables  such  that  for  any  integer  n  >  1,  any  set 

{tj  <  t2  <  •  •  •  <  t^}  of  parameter  values,  and  any  real  number  a. 
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(4. 1) 


P  [Z  <  a|Z  , 
n  n-1 

If 

P[Z  <a|  .  .. 

n  n-1 


Z^  ,  Z  ]  =  P[Z  <  a|  Z  ] 
2  1  n  n-1 


ZJ  =  P[Z  <a|Z^  . 

1  n  n-1 


Z^  ]  ,  n  > 
n-  |JL 

(4.2) 


then  {Z^}  is  called  a  Markov  process  of  order  p..  By  definition,  if  the 
term  Markov  process  is  used  without  any  qualifications,  the  order  is 
unity. 

A  ra,ndom  process  is  called  a  discrete  or  a  continuous  parameter 

process  according  to  whether  the  index  set  7  is  coulRable  or  not.  A 

random  process  is  called  stationary  if  for  any  t^t7,  i  =  1,  .  .  .  ,  n, 

and  any  reala  such  that  (t.+ a  )eT,  i  =  1,  •  .  .  ,  n,  the  joint  distribution 

of  the  random  variables  Z  •  •  ■  *  •  >  Z  is  independent  of 

l+a  ^2+q  n+Q 


of  for  all  finite  n.  The  parameter  t,  sometimes,  will  be  identified  with 
the  physical  variable  time. 

If  the  random  variables  Z^  of  a  Markov  process  can  assume  values 

only  in  some  countable  set  D,  the  process  is  called  a  Markov  chain; 

the  chain  is  said  to  be  finite  or  infinite  according  as  O  is  finite  or 

infinite.  If  O  is  taken  to  be  a  collection  of  modes  *  D.,  the  time  develop- 

J 

ment  of  a  particular  sample  function  of  a  discrete  parameter  Markov 
chain  can  be  thought  of  as  an  evolution  through  the  modes  D.  e  D: 


.  .  .  ,  D.  ,  D.  ,  D.  ,  .  . 

h  h  h 


For  a  discrete  parameter  Markov  chain  (Z^,  n  =  1,  2,  .  .  .},  given 
an  initial  mode,  the  time  development  is  completely  described  by  a 
set  of  transition  probabilities 

p..(n)  =  P[Z  =  D,|  Z  ,  =  D.] 

'  '■  n  j '  n-1  i-* 


The  standard  terminology  of  Markov  chain  theory  calls  for  the 
use  of  the  term  "state"  rather  than  "mode.  "  The  term  "state"  is  not 
used  here  to  prevent  confusion  with  the  state  (vector)  of  the  differential 
system. 
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i  • 


specified  for  all  n  >  1,  and  all  D^,  &  D.  Of  course, 

Pij  (n)  >  0  and  ^  p.j(n)  =  1 

j 

where  the  summation  extends  over  all  modes  in  D.  The  matrix 


P(n)  =  (Pij(n)) 

is  called  a  Markov,  or  stochastic,  or  probability,  or  transition,  matrix. 
Such  a  matrix  together  with  a  starting  distribution 


a.  =  P[Z,  =  D.l 
J  1  J-* 


completely  specifies  the  chain.  If  p^^  is  not  a  function  of  n,  the  chain  is 
called  homogeneous. 

Henceforth,  unless  explicitly  indicated,  the  term  Markov  chain 
will  be  used  to  denote  finite  homogeneous  chains  only. 

If  the  elements  of  the  kth  power  of  the  transition  matrix  of  a  Markov 


chain  are  denoted  by  p.\ 


(k) 


then 


pW  =  P(2,,=  D.  I  2^1  =  Oil, 


k  >1; 


,(k) 


that  is,  p^\  '  is  the  transition  probability  of  going  from  mode  D.  to  mode 
D.  in  k  steps. 

J  (k) 

If  for  some  finite  k  >  1,  p^^  >  0,  then  it  is  possible  to  go  (i.  e.  , 

with  nonzero  probability)  from  mode  i  to  mode  J  (the  modes  will  often 

be  referred  to  by  their  subscripts  only),  and  this  will  be  denoted  by 

(k)  (k) 

i  If  there  exists  no  finite  k  >  1  for  which  p,.  >0,  that  is,  p'.  '  =  0 

—  ij 

for  all  k  >  1,  then  the  transition  from  i  to  j  cannot  occur  and  this  will 
be  denoted  by  i-y-^j. 
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A  trainiient  set  V  of  mode*  ii  the  set  of  all  modes  i  to  which 
there  correspond  modes  j  such  that  and  A  nonempty  set  JC 

of  modes  is  called  closed  if  for  every  i  t.  JC  and  every  j  ^  K.  i  j. 

If  the  jth  row  and  jth  column  of  some  Markov  matrix  are  deleted  for 
all  j  ^  i/C  where  JCi*  some  closed  set,  the  matrix  remaining  after 
deletion  is  still  a  Markov  matrix.  A  mode  i  is  called  absorbing  if  and 
only  if  =  1;  if  a  closed  set  contains  only  one  mode  then  this  must 
be  an  absorbing  mode.  A  closed  set  £.  of  modes  is  called  ergodic  if 
i — >j  for  all  i,  j  1 6, ,  i.  e. ,  is  a  closed  set  no  proper  subset  of  which 
is  closed.  Every  finite  Markov  chain  contains  at  least  one  ergodic  set; 
the  complement  of  the  union  of  all  the  ergodic  sets  is  the  traxisient  set. 
If  the  set  of  all  modes  of  a  chain  is  ergodic,  then  the  chain  is  called 
ergodic.  A  regular  chain  is  an  ergodic  chain  such  that  for  all  i,  j, 
p|j  >  0  for  all  n  >  N  for  some  N.  Every  ergodic  set  can  be  parti¬ 
tioned  into  classes  Cy  *  *  *  •  ^  such  that  every  one-step  transi¬ 
tion  carries  the  chain  from  a  mode  in  to  a  mode  in  a 

mode  in  ^  ^  to  a.  mode  in  C  ^  ,  from  a  mode  in  to  a  mode  in 

C.  that  is,  the  system  moves  cyclically  through  the  classes  .  . .  , 

C  ^  2’  "  ‘  ^  d'  ^  1*  •  *  • .  integer  d  is  called  the  period. 

If  d  =  1  then  the  ergodic  set  is  called  aperiodic;  if  d  >  1  then  the  set  is 
called  cyclic.  It  follows  that  if  p|^^  >0,  i  a  then  n  is  an  integer 
multiple  of  d.  It  can  be  shown  that  there  exists  an  integer  b  with  the 
property  that  for  all  i  a  £>,  ^  0  for  all  k  >  b. 

By  a  reniunbering  of  the  modes  of  a  Markov  chain,  the  transition 

31 

matrix  P  always  can  be  written  in  the  form 


(  4.  3) 
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E.,  E,,  E  are  thMnielvei  Markov  matrices  and  correspimd  to 

tke  ergodic  sets  6  £.  2>  *  •  *  •  ^  chain.  The  matrix  T 

corresponds  to  the  transient  set.  By  a  further  reordering,  any  of 

31 

the  Markov  matrices  Ej^  can  be  written  as 


(4.4) 


where  p 


ij 


is  an  element  of  the  matrix  C  if 

n 


i  a  ^ 


4.  2  ith  Moment  Stability 

As  in  Chap.  Ill,  the  differential  system  is 

X  S' Aj^x,  ^-l  —  ^  ^  k  =  1,  2,  ...  (4.  5) 


which,  as  shown  in  Eqs.  (3.1)  to  (3.6.),  may  be  reduced  to' 

%  =  Jkik-i  •  • -ii-o 

In  this  Chapter  the  matrix  sequence  ^  Markov  chain 

with  m  possible  modes  X^,  . . . ,  X^,  an  initial  probability  distribu¬ 
tion  {a^}  on  the  modes,  and  an  m  x  m  constant  transition  matrix 
P  .  (Py): 

•,=  pir,  =  x,]  Pu>*iik  =  *ji5k-i  =  *i] 

It  is  required  to  evaluate  explicitly  all  the  moments  of  the  output 
vector  process  in  particular,  the  limiting  behavior  as  k->ao. 
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i kVi  =  ikJk-iVr  ^  *k-z  *k-i  •” 

■erved,  at  aay  sample  poiat  u,  they  will  deliaeate  a  subset  ^  ^  of  the 
range  /t  =  {X^.  •  •  • ,  X^}  in  which  must  assvime  its  value. 
in  general,  is  not  coincident  with^.  Since  1*  &  firat>order 

Markov  chain,  the  distribution  of  1*  conditioned  on  the  subset 
Hence  <ik>  is,  in  general,  a  Markov  process  of  order  >  1.  More¬ 
over,  the  range  of  {xj^}  is  countable.  So  finally,  {x^^}  is  a  homogeneous, 
nonstationary,  infinite  Markov  chain  of  order  >  1. 

Taking  the  expectation  in  £q.  (4.4  ), 

Ex^  -  =<5k-  -5i>=‘o 

which  by  the  dei.'ivation  in  Appendix  D  reduces  to 

EXk  = 

where  is  the  direct  sum  of  the  X^: 

Yl=X^  Q  •••« 
is  the  square  matrix  of  order  mn: 

=  P  9  =  nth- order  identity  matrix; 

Ct^  is  the  mn  x  n  matrix: 


and  is 


the  n  X  mn  matrix: 


'  y 

m  identities 


V 
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Equation  (4.  7)  give*  the  expected  value  of  the  sequence  To  obtain 

the  ith  moment,  take  the  ith  self-Kronecker  product  of  Eq.  (4.  6)  with 
itself  to  obtain  (as  derived  in  Eqs.  (3.  9)  -  (3. 12)) 


^[i]  =  5k[i]-*-5i[i]*o[i] 

Hence 

®*k[i]  ^  ^i^^l^i^  ^i^i*0[i] 

where 

^i=^l[i]  Vi]  ' 

Qi=P«i„i  . 


‘n 


La, 


IK 


m  identities 

If  the  Markov  chain  specialises  to  an  independent  process,  Eq.  (4.8  ) 
gives  the  results  obtained  in  the  previous  chapter,  as  is  shown  in  Appen¬ 
dix  E.  From  Eq.  (4.8),  by  using  arguments  similar  to  those  in  the 
previous  chapter,  sufficient  conditions  for  stability  and  asymptotic  sta¬ 
bility,  in  the  1th  moment,  can  be  found;  that  these  conditions  are  not 
necessary  is  demonstrated  later. 

THEOREM  1:  (A)  The  trivial  solution  of  (4.  5)  is  \miformly  stable 

T 

in  the  ith  moment  if  has  no  eigenvalues  outside  the  unit  circle. 
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and  any  eigenvalue  on  the  unit  circle  has  index  one.  (B)  It  ia  uniformly 

aaymptotically  (in  fact,  exponentially  asymptotically)  stable  in  the  ith 

T 

moment  if  all  eigenvalues  of  lie  inside  the  unit  circle. 

PROOF:  (A)  By  the  assumptions  on  the  location  and  the  index  of 

the  eigenvalues,  it  follows  from  the  Lemma, Chap.  Ill,  that  every  element 
T  k 

of  the  matrix  )  is  bounded  xiniformly  in  k.  Hence 

II  ^ i**  bounded  uniformly  in  k,  i.  e. ,  there  exists  an 

M  >  0  such  that  M  =  sup  ||  ^.(Y.Q^)^Y.  CL.  ||  .  Given  any  c  >  0, 

k_>l  111  11 

choose  =  <  /M.  From  £q.  (4.8) 

“  ll*0[i]U  II  i]  11  i*  less  thane.  But||xQjjj||  <  6^ 

is  equivalent  to  ||  Xq|}  <  6  for  some  6  >  0.  Also  4^,  and  hence  6,  do 
not  depend  on  tQ  because  the  starting  probability  distribution  is  taken 
to  be  independent  of  tg.  Hence  the  trivial  solution  is  uniformly  stable 
in  the  ith  moment. 

T 

(B)  Since  all  eigenvalues  of  Y^Q^  have  moduli  less  than  one,  it 

follows  by  the  Lemma,  Chap.  Ill,  that  (Y.qT)^^— ^  0  as  k-^oo.  Hence 
T  k  *  ^ 

'^i^^i^i  ^  k— >OD.  Note,  moreover,  that  the  convergence 

to  the  aero  matrix  is  independent  of  t^  for  the  Markov  chain  is  assumed 
to  be  homogeneous  with  an  initial  distribution  that  is  independent  of 
tg.  Therefore,  given  any  6  >  0,  it  follows  from  Eq.  (4.  9)  that  for  all 
tq,  II  Xql  <6  implies  as  k— >co  uniformly  in  tq,  and  uni¬ 

formly  on  II  Xqll  <4  (since  the  system  is  linear).  Hence  the  trivial 
solution  is  uniformly  quasi-asymptotically  stable  in  the  ith  moment. 
Moreover  by  (A)  it  is  uniformly  stable'  in  the  ith  moment.  Therefore 
it  is  uniformly  asymptotically  stable  in  the  ith  moment. 

To  show  exponential  asymptotic  stability  in  the  ith  moment,  take 
norms  in  Eq.  (4.8)  to  obtain 
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llE*^[i]ll  <  II  3ill  II  II 

and  then  use  the  fact  that  ||  (Y.Q^)  || 

0  as  k— >00.  (See  Theorem  2,  Chap.  III.  ) 


converges  exponentially  to 


DEPENDENCE  ON  STRUCTURE  CF  MARKOV  CHAm.  The  stability  proper¬ 
ties  of  the  trivial  solution  are  independent  of  the  ordering  of  the  modes. 
They  are,  however,  dependent  on  the  transient  and  ergodic  set  struc¬ 
ture  of  the  chain  and  on  the  initial  distribution  of  the  modes.  To  demon¬ 
strate  this  dependence,  it  will  be  convenient  to  assume  that  the  transi¬ 
tion  matrix  P  of  the  chain  is  in  the  canonic  form  (  4.  3)  with  the  E  matrices 
in  the  form  (4.4  ).  In  the  stability  study  of  a  particular  system,  the 

availability  of  P  in  a  canonic  form  also  facilitates  the  computation  of 

T 

the  eigenvalues  of  Y.Q.  ,  and  further,  by  exhibiting  explicitly  the  tran¬ 
sient  and  ergodic  sets  aids  in  the  understanding  of  the  problem  under 
consideration.  In  practice,  of  course,  these  advantages  may  be  negated 
partially  by  the  labor  required  to  reduce  the  given  transition  matrix  to 
a  canonic  form. 

Suppose  that  P  is  in  a  canonic  form.  Also  for  algebraic  convenience , 
assume  that  there  are  two  ergodic  sets  ^  transient  set  HT . 

Then  Q.  assumes  the  form 

‘  Pi  ®  ’ni  "  0  1 


Q.  =  P.  9  I  i  = 
1  i  n* 


^2  ®  'ni 


(4.10) 


Us,  ®  I  i  S-  ®  I  i  T  ®  I  i-J 
1  ^  n*  2  n*  n* 


Let  U,,  U, . U  ,  be  the  modes  of  let  V,,  V_ . V 

1  m^  1  1  £  ^2 

be  the  modes  of  and  let  W, ,  W_,  .  .  .  ,  W  be  the  modes  of 

£,  1  c,  m. 


-41- 


Define 


n  '  =  “l[il  •  •  • 

«...9 

yJS)  =  Wm,e...9 

Then,  by  definition  of  Y^, 

Y^  =  Yf >  •  Yf'  • '  yP>  , 

whence 


fS”  (E,^  *  V  > 

0 

0 


0 


y!2),e 
1  ' 


T 

2 


®  V) 


0 


’'f*  *  V  >■ 

Y<">(SJ  •  I^i) 

YP’d'"'  •  y) 


(4.11) 

The  structure  of  the  matrix  on  the  right-hand  side  of  the  above  equation 

shows  that  the  eigenvalues  of  (E^  ^  ),  of  y|^^  (eJ  9>  ), 

(3)  T  T 

and  of  Y^  (T  9  1^^)  are  the  eigenvalues  of  .  The  computation 

of  the  eigenvalues  of  lui*  been  reduced  to  the  computation  of  the 

eigenvalues  of  lower-order  matrices. 

If  E.  is  a  cyclic  set,  some  simplification  in  the  determination  of  the 

M  \  T 

eigenvalues  of  Y^  '  (E^  ^  I^^  )  is  possible.  This  is  shown  in  Appendix 

F. 

If  the  initial  distribution  {a^}  assigns  sero  probability  to  all  modes 
not  in  (or  indeed,  in  any  closed  set  of  modes),  then  the  system  be¬ 
havior  must  be  independent  of  the  values  of  p^,  X^,  for  all  i»  j  ^ 

For,  S  1*  A  closed  set;  hence,  by  the  choice  of  the  initial  distribution. 
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the  system  can  never  enter  any  mode  not  in  In  the  case  of  the 

ith  moment  this  result  can  be  obtained  algebraically  by  noting  that  if 

T  It 

a^  =  0  for  all  j  ^  then  hence  Exj^  ^  via  Eq. 

(4.  8  ),  is  for  all  k  independent  of  X^,  for  all  i,  j  ^  S  y 

In  addition  to  assuming  that  {S'^}  assigns  zero  probability  to  all 

modes  not  in  £*.,  suppose  further  that  every  eigenvalue  of 
(11  T  ^ 

Y;  '(E,  (9  I  i'')  is  inside  the  unit  circle.  Then  the  trivial  solution  is 
1  1  ^  a 

asymptotically  stable  in  the  ith  moment.  But  by  proper  choice  of 

E^.  T.  the  matrices  •  I„i)  and  «  I^i). 

T 

and  hence  the  matrix  Y^Q.  ,  certainly  can  have  eigenvalues  outside  the 

unit  circle.  Hence  the  converse  of  the  preceding  theorem  is  false; 

namely,  for  the  trivial  solution  to  be  asymptotically  stable  in  the  ith 

T 

moment,  it  is  not  necessary  that  all  eigenvalues  of  Y^Q.  have  moduli 

less  than  unity.  Likewise,  if  the  trivial  solution  is  stable  in  the  ith 

T 

moment,  it  is  not  necessary  that  no  eigenvalue  of  Y^Q.  lie  outside  the 

unit  circle  and  that  any  eigenvalue  on  the  unit  circle  have  index  one. 

(11  T 

Suppose  now  that  all  eigenvalues  of  Y'  '(E,  9  ^  i) 

(3)  T  in 

Y'  '(T  fl)  I  {)  are  inside  the  unit  circle  and  that  the  elements  of  the 
n* 

matrices  are  such  that  for  some  initial  distribution  the  trivial  solu¬ 
tion  is  unstable  in  the  ith  moment.  (This  could  occur  if  all  elements 
of  all  the  matrices  V.,  j  =  1,  .  .  .  ,  m.  are  greater  than  one.  )  By 
choosing  an  initial  distribution  which  assigns  zero  probability  to  all 

modes  in  that  is,  by  choosing  a  a  a  _ 

^2’  ’  '  ®  m^+l’  mj+2’  ’  m^+m^  ^  0, 

the  trivial  solution  becomes  stable,  in  fact,  asymptotically  stable,  in 
the  ith  moment.  This  shows  that  stability  in  the  ith  moment  and 
asymptotic  stability  in  the  ith  moment  depend  on  the  choice  of  the  ini¬ 
tial  distribution  {a.}. 

1 

It  is  a  fact  that  with  probability  one  the  chain  remains  in  the  tran¬ 
sient  set  through  only  a  finite  number  of  transitions.  Yet  stability 
in  the  ith  moment  and  asymptotic  stability  in  the  ith  moment  depend, 

in  general,  onp..,  X.,  i  For,  consider  the  simplest  possible 
ij  i 
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problem  of  studying  the  first  moment  stability  of  a  first-order  system 
with  one  absorbing  and  one  transient  mode.  The  P  and  matrices 
are  assumed  to  be 


and 


.^k  _k 

X,  T  -J 


(  4. 12) 


where  h^^  is  a  function  of  X^.  S,  and  T.  Using  Eq.  (4.8  )  to  evalu¬ 
ate  Exj^, 

E*k  ^^1^1  *2\-1^2  ^2^2^  ^  *0 


which  clearly  depends  on  S,  T,  X^i  and  {a^,  a^}. 

The  results  of  the  preceding  four  paragraphs  show  that  if  the  chain 
contains  more  than  one  ergodic  set  and  possibley  a  transient  set,  then 
it  may  be  possible  by  changing  the  initial  distribution  on  the  modes  to 
make  a  stable  trivial  solution  unstable  and  vice  versa.  Suppose  now 
that  the  chain  is  ergodic,  that  is,  there  is  only  one  ergodic  set  and  the 
transient  set  is  empty.  The  question  arises  as  to  whether  (a)  the 
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convergence  of  divergence  of  Ex^  with  k  depends  on  the  initial  dis¬ 
tribution  of  modes;  (b)  the  converse  of  the  preceding  theorem  is  true, 
namely,  is  it  possible  to  ensure  stability,  and  asymptotic  stability,  in 

the  ith  moment,  by  knowing  the  location  and  index  of  the  eigenvalues  of 

T  ♦ 

,  if  the  chain  is  ergodic.  The  second  question  is  not  answered 

here.  A  partial  answer  to  the  first  question  is  given  by  the  following 

example  which  shows  that  the  value  of  Ex^^^^j  for  large  k  can  depend 

markedly  on  the  initial  distribution. 

The  example  is  as  follows:  The  system  is  of  order  two,  has  two 

modes,  and  tj^-tj^  1  "  ^  ^ 

the  two  values 


>  1.  The  random  matrix  can  assume 


-P 


a  fixed,  p  =  2-s  or  x. 


Hence  the  two  respective  values  of 
Xj  =  e“l2  and  X2  =  -e®!^ 


where  I  is  the  nth- order  identity  matrix.  Let  the  transition  matrix 
n 

of  the  chain  be 


=  (e^L  ® 


®  I2)  ^  (e“LP'^)  9  I2 


(4.13) 


(4. 14) 


The  answer  to  the  second  question  is  in  the  affirmative  if  the 
chain  is  regular  and  all  elements  of  X.  are  nonnegative.  This  case 
will  be  considered  in  more  detail  in  a'^ later  report. 
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where 


Equation  (4.14)  follows  from  (4. 13)  by  the  commutative  property 
(3. 11)  of  the  Kronecker  product.  Using  this  commutative  property 
again, 

(Y^Qj^)^  =  ({e'^LP'^)  ®  l2)((e“LP'^)  ®  I2) 

=  (e‘"LP'^)^  ®  I2  =  e^“(2p-l)l2  ®  h 

It  follows  that 

=  e^^*{2p.l)^I^ 
whence  by  Eq.  (4.  8) 

(4. 15) 

Evaluating  Ex2j^^^  in  a  like^fashion, 

Ex2j^=  e2^*(2p-l)\  (4.16) 

Hence  if  j  is  odd,  Ex^  depends  on  the  initial  distribution  {a^,a2};  if 
j  is  even,  it  does  not.  Equations  (4. 15)  and  (4. 16)  show  that  the  trivial 
solution  is  stable  in  the  first  moment  if 

I  e’V/I^  I  5  1  (4. 17) 

A  X 

(Note  that  +  e  i/2p-l  are  the  eigenvalues  of  TfjQj  which  checks  with 

the  results  of  Theorem  1.  )  Equations  (4. 15)  and  (4. 16)  also  show  that 

(4. 17)  is,  moreover,  a  necessary  condition  Ant  first  moment  stability. 

Likewise,  (4. 17)  changed  to  a  strict  inequality  gives  the  necessary 
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and  fufficient  condition  for  asymptotic  stability  in  the  ith  moment. 

In  exactly  the  same  fashion  it  can  be  demonstrated  that  the  trivial 
solution  is  stable  (asymptotically  stable)  in  the  ith  moment  if  and 
only  if  I  e'^-s/Tpir  |  <  1  (e^“^y2p^l  <  1). 

4.  3  Almost  Sure  Asymptotic  Stability 

As  in  the  independent  case,  there  exists  the  following 

THEOREM  2:  The  trivial  solution  is  equiasymptotically  stable 

T 

almost  surely  if  for  some  even  i,  all  eigenvalues  of  are  inside 

the  unit  circle.  (Compare  with  Theorem  3,  Chap.  III.  ) 

PROOF:  The  proof  is  virtually  identical  to  that  of  Theorem  3  of 
Chap.  III. 

By  Eq.  (4.8), 

Summing  on  k, 

00  oo 

k=l  ksl 


=  Sai 


1  mn* 


because  of  assumption  on  the  eigenvalues  of  . 

The  remainder  of  the  proof  is  exactly  the  same  as  that  of  Theorem 
3  of  ChapJlI. 

The  converse  of  the  above  theorem  is  false  as  is  shown  by  Example 
1  of  ChapJII. 


4.  4  Example 

Consider  the  same  system  as  in  Example  1  of  Chap.III  but  with  a 
Markov  structure: 

can  assume  two  possible  values,  b,  <  0  and  b.  >  0;  t,  -t,  ,  =  T 

'  1  —  2  —  k  k-1 


-47- 


b  T 

for  all  k.  Hence  can  assume  the  two  possible  values  K,  =  e 
b^T 

and  X2  =  e  The  transition  diagram  for  the  Markov  chain  is 


Since  the  system  is  of  order  1,  =  P,  and 


T 

Since  all  entries  of  the  matrix  are  positive,  it  can  be  shown 

that  a  necessary  and  sufficient  condition  for  the  asymptotic  stability 

T 

in  the  ith  moment  is  that  every  eigenvalue  of  Y.Q.  lies  inside  the  unit 
circle.  Solving  for  the  eigenvalues 

I  -  xijl  =  -  MP.xj  t  P^x‘  )  MI  -  p,  -  p„)xjx‘  =  0 

The  Schur-Cohn  criterion  can  be  used  on  this  polynomial  in  X.  to 
obtain  Figs.  4-6  which  show  the  regions  of  stability  in  the  Pg~P^ 
plane.  The  line  segment  labeled  "independent"  in  the  figures  is 
the  set  of  points  (p^,  p^)  such  that  the  Markov  chain  becomes  an  in¬ 
dependent,  identically  distributed,  random  sequence. 


-48- 


I 


fixed 

increasing 
from  1  to  oo 


* 


I 


4.  5  Some  General  Comments 


In  both  Chaps. Ill  and  IV,  the  fact  that  the  matrices  are  of  the 
form  'exp[B]  has  not  been  made  use  of.  Hence  the  results  pertain 
not  only  to  piecewise  constant  differential  equations  but  also  to  all 
random  linear  difference  equations  x^  s  ^  ,  where  {  is  an 

independent,  identically  distributed,  rauidom  process,  or  more  gener¬ 
ally,  ^a  finite  Markov  chain. 

Such  difference  equations  arise  in  the  study  of  randomly  sampled 
linear  systems,  whose  second  moment  stability  has  been  investigated 
by  Kalman.^  Kalman  considers  the  case  where  the  successive  aam- 
pling  intervals  are  independent,  identically  distributed,  random 
variables.  This  corresponds  to  the  situation  when  inde¬ 

pendent,  identically  distributed,  random  process.  Kalman's  method 
consists  of  taking  the  expectation  of  the  scalar  product, 


=  E  <  ^  .  .  JjXq,  ^ 
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which  lead!  him  to  the  use  of  power  product  matrices.  The  method 
is  not  as  straightforward  as  the  one  presented  here  and  suffers  from 
the  limitation  that  it  cannot  be  directly  extended  to  the  study  of 
moments  of  order  >  2  ,  and  to  other  statistical  structures  such  as 
Markov  chains. 

All  of  the  material  presented  in  this  section  can  be  extended  to 
the  case  where  {  is  a  multiple -order  Markov  chain.  This  ex¬ 
tension  will  be  included  in  a  later  report. 

V.  THE  FORCED  LINEAR  SYSTEM 

Summary 

For  random  linear  systems,  a  theorem  is  proven  which  gives 
sufficient  conditions  on  the  input  and  on  the  solutions  of  the  unforced 
system,  in  order  that  solutions  of  the  forced  system  be  bounded  in 
the  mean  norm.  The  results  of  the  theorem  are  immediately  appli¬ 
cable  to  the  piecewise  constant,  linear  system  considered  in  Chaps. 

Ill  and  IV. 

The  preceding  chapters  consider  the  stability,  with  respect  to 
changes  in  initial  conditions,  of  a  fixed  solution  function.  Hence,  the 
input  to  the  system  is  necessarily  assumed  to  be  fixed,  usually  at 
zero.  In  practice,  however,  it  is  the  behavior  of  the  system  in  the 
presence  of  any  input,  belonging  to  some  class  of  possible  inputs, 
that  is  of  interest. 

For  deterministic  linear  systems  a  common  definition  of  sta¬ 
bility  is:  the  output  to  every  bounded  input  is  bounded.  Because  of 

19  33 

linearity  this  definition  of  stability  is  equivalent  ’  to  any  of  the 
following  properties  of  the  unforced  linear  system:  (i)  the  trivial 
solution  is  uniformly  asymptotically  stable;  (ii)  the  trivial  solution 

t 

is  exponentially  asymptotically  stable;  (iii)  ^  |  W(t,  t)!  dr  is 
bounded  in  (t,  t^),  t  >  t^  , 


where  W(t,  t)  ia  the  fundamental  mat;rix. 

The  following  theorem  partially  generalises  some  of  these  results 
to  the  random  case: 

THEOREM  1:  Consider  the  vector  differential  equation 
X  =  A(t)x -f  b(t), 

where  both  {A(t)}  and  {b(t)}  are  random  processes  and  sup  ||  A(t)||  is 

a.  a.  bounded.  For  almost  every  u> ,  assume  that  A  and  b  are  suffi- 
35 

ciently  smooth  so  that  solutions  of  (5. 1)  exist  and  are  unique.  Let 
W(t,  T ),  t,  T  >  0,  be  the  fundamental  matrix  of  the  homogeneous  equa¬ 
tion  (b  ■  0)  which  is  normalized  to  W(t,  t)  =  I,  t  >  0.  Denote  the  solu¬ 
tion  bf  (5. 1)  by  g^(t,  Xq,  Iq).  Now  consider  the  following  propositions 
(M^  are  finite  positive  constants  independent  of  (t,  t^)): 

(a)  The  trivial  solution  of  the  homogeneous  equation  is  exponen¬ 
tially  asymptotically  stable  in  the  mean  norm;  or  equivalently,  there 
exist  two  constants  K,  v  >  0  such  that 

E II  W(t ,  tg)  I  <  K  exp  [  -v(t-to)]  ,  t  >  tp. 

Ell  W(t,T)|j  dT  <Mj,  t  >tQ  . 

.  a.nd  {b(tU  in  statisticallv  independent  of 


(b)  y 
*0 

^c\  Tfirl 


I 


Propolition  (a)  states  that  the  unforced  system  behaves,  in  the 
mean  norm  sense,  like  an  unforced,  constant  coefficient  linear  system 
all  of  whose  characteristic  roots  have  negative  real  parts. 

Propositions  (c)  and  (c*)  are  stochastic  equivalents  of  the  oft- 
heard  definition  of  stability  for  deterministic  linear  systems;  to 
every  bounded  input  there  corresponds  a  bounded  output. 

The  theorem  gives  a  partial  justification  for  considering,  in  sta¬ 
bility  studies  of  random  linear  systems,  the  simpler  unforced  system 
instead  of  the  whole  family  of  forced  systems.  This  simplification,  of 
recognised  value  in  the  study  of  deterministic  linear  systems,  is  perhaps 
of  even  greater  value  for  random  linear  systems  because  of  the  more 
complicated  nature  of  the  problem. 

Before  proving  the  theorem,  consider  the  following 

LEMMA:  Proposition  (b)  of  Theorem  1  implies  that 
l|W(t,tQ)i  <M^.t>tQ>0. 

19 

PROOF:  The  proof  given  below  follows  the  one  given  by  Kalman 
for  the  deterministic  case. 

The  fundamental  matrix  satisfies  the  homogeneous  differential 
equation:  ‘ 

^  W(t,T)  =  A(t)W(t,T) 

Hence 

^  W-\t,  T  )  =  -  W^t,  T  )|p  W(t,  T  )W‘ V.  T  ) 

=  -W'V.T)A(t)W(t,T)WV,  t) 

=  -WVT)A(t) 

On  interchanging  t  and  t, 

^W"\T,t)=  -W’^T,t)A(T) 
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from  which  it  follows  that 


w(t,  T)  =-W(t,  t)A<t) 


,-l, 


since  W  (r.t)  =  W(t,  t).  Integrate  with  respect  to  t,  use  the  fact 
that  W(t,  t)  =  I,  and  then  take  norms: 


W(t,tQ)  -  I  =  J*-W(t,  T)A(T)dT 


Till  II  A(t)  II  dr 
t 


II  W(t.t  )  -  Ill  <  f  11  W(t, 

<  [a.  s  .  sup(sup  II  A(t)||  )]  f  II  W(t,  t)  II  dr  . 

Now  II  W(t,tQ)||  <  II  W(t,  tp)  -  l||  +  II  l||.  H^^nce,  upon  taking  expectations, 
E||W{t,t  )||  <[a.  s.  sup  (sup  II  A(t)||  )]  E  ^Hwit,  t)1|  dx  +E||i|| 

t>  0 


<  [a.  s.  sup  (sup  II  A(t)ll  )]  M.  +  ||  l|| 

t>0  ^ 


Hence  E  II  W(t,  tp)  II  is  bounded  in  (t,  t^),  t  >  tQ- 

PROOF  of  Theorem  1:  To  show  (=»)-— ^(V») 

t  ^  • 

J  E||W(t,T)||  dT  <  r  Ke" dx  <  ^ 

To  show  (b)^=^(c)  .  The  solution  of  (5. 1)  can  be  expressed  as 

gb(t  .Xq '*0^  *  J  W(t,x)b(x)dx 
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which  upon  taking  norms  becomes 

||gb(t.X„.t„)|l  <  II  J  |W(t,T)|l  |lb(T)l|  dT 

(5.  2) 

Since  {iS(t)}  and  {b(t))  are  independent, 

E|l8b<'’V‘0>ll  ^ElIWU.Vll-  llxol 

t 

+  J  E||  W{t.  t)|1.E|1  b(T)||  dT. 

^0 

By  the  preceding  Lemma,  E  ||  W(t,  t^)  1|  <  M  and  by  hypothesis, 
E||b(T)ll  <^2-  Hence 

‘0 

<  M  ^  11  Xq  II  +  M  2^1 

=  M3(Mj,  1|xq|1  ) 

To  show  (b)=^{c’).  From  inequality  (5.  2) 

l|gb(t.*o'Vll  ^  II  W(t.tQ)l|  •  llxjl 

t 

+  [a.  8.  sup  (  sup  II  b(t)l|  )]  f  11  W(t,  t)|1  dT 

t>0  Jtp 


Therefore, 


E||gb(‘.Xo.‘o>ll  <E||W(t,t„)||-  ||X(|||  +M^ 


dT 


whence  by  the  preceding  Lemma,  and  by  (b) 

E|lgb(‘-VVll  ll^oU  +^4^1 

=  MgtMj,  HxqII) 
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The  results  of  the  preceding  theorem  are  immediately  applicable 
'  to  the  random  linear  differential  system  considered  in  Chaps.  Ill  and  IV. 

The  vector  differential  equation  now  becomes 

‘  X  =  Aj^x  +  b(t),  <  t  <  tj^  ,  k  =  1,  2,  .  .  .  (5.  3) 

Suppose  that  the  trivial  solution  of  the  homogeneous  system  is  exponen¬ 
tially  asymptotically  stable  in  the  ith  moment  for  some  even  i.  Then, 
as  shown  in  Chap.  II,  it  is  exponentially  asymptotically  stable  in  the 
mean  norm.  Hence,  by  the  theorem  just  proven,  it  follows  that 
If  for  the  differential  system  (5.  3),  the  trivial  solution 
of  the  homogeneous  system  is  exponentially  asymptotically 
stable  in  the  ith  moment  for  some  even  i,  then 
E  II  gjj(t,  Xq,  tp)  II  is  bounded  uniformly  in  (t,  tjj)  for  all 
bounded  x^,  provided  either 

{b(t)}  is  statistically  independent  of 

^)}  and  E  ||  b(t)  ||  is  hounded  uniformly 

•  in  t, 

or 

{b(t)}  is  a.  s.  bounded. 

APPENDIX  A 

KRONECKER  AND  POWER  PRODUCTS  OF  MATRICES* 

In  the  definitions  and  identities  that  follow,  except  for  finiteness, 
no  assumptions  about  the  dimensions  of  the  matrices  are  made  unless 
indicated  explicitly,  or  implied  by  the  use  of  operations  such  as  in¬ 
verse,  trace,  determinant,  the  conventional  sum  A  +  B  and  product 
AB,  etc. 

KRONECKER  PRODUCTS 

V  If  A  and  B  are  matrices  of  order  a  and  p,  respectively,  of  the 

transformations 

■  iti 

,  For  a  concise  statement  of  the  properties  of  these  products, 

see  MacDuffee,  Chap.  VII. 
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« 


= 


y  a..?.  , 
L  ijj 
j=l 


’ll  =  I  ’’ii’'!  ■ 

j=l 


(A.1) 


the  product  vector  (column) 

(§1  ’ll  .  ^1^2  ‘  ■  •  •  '  ’’p  *  ^2  "’l  ’  ^2  ^2  ’  •  •  ■  ’  ^2^^^  ’  •  ■  '  *  ^ 

is  a  linear  transform  of  the  corresponding  unprimed  product  vector 
(column) 

51^2’ •  ‘  • ’^I’^p  *  ^2^1’  ^2'^2’”  ’  ^2’^p . ^Q^p^  (A.  3) 

where  the  linear  transformation  is  given  by  the  Kronecker  (or  direct) 
product  matrix: 


A  ®  B  = 


■  "11® 

"iQ 

"21® 

*22® 

"2q 

-  "ol® 

*.2® 

a 

a  a 

B 


DEFINITION:  Given  an  o,  x  a_  matrix  A  =  (a..)  and  a  p,  x  p. 

1  2  '  ij'  ^1^2 

•ix  B  =  (b..),  the  K 
ij. 

OjPj^  X  a  2^2 


matrix  B  =  Kronecker  (or  direct)  product  of  A  by  B  is  the 


A  <»  B  -  (a..B)  = 


"11® 


a  ,B 

Oil 


^1n  B 
^"2 


a  B 
"l"2 


The  product  is  associative 

(A  ®  B)  9  C  =  A  9  (B  «  C) 


« 


-5  7. 


from  which  follows  the  definition  of  the  ith  self-Kronecker  product: 


=  A  ®  ®  A 

to  i  factors.  It  obeys  the  relation 

In  fact,  the  more  general  identity 

AjB^  ®  A2B2  ®  .  .  .  ®  A.B.  =  (A^  ®  A^  ®  .  .  .  ®  A.)(B^  ®  B^  ®  .  .  .  ®  B.) 

holds,  from  which  it  follows  that  if 

Ax.  =  X.  .X.  By.  =  u.  y. 

Ill  'j 

then 


(A  ®  B)(x.«y.)  =  Axj  ®  By^  =  X  .x.  ®  y^  =  ®  Yj)  » 

that  is,  if  X^  and  are  eigenvalues  of  A  and  B,  respectively,  with 
corresponding  eigenvectors  x^  and  y^,  then  A  ®  B  has  the  eigenvalue 
X^Pj  with. the  associated  eigenvector  x^  ®  y^. 

Most  of  the  following  identities  are  readily  verified: 


(a)  (A  +  B)  ®  C  =  A®C  +  B  ®  C 

(b)  (A  ®  B)^  =  A*^  ®  B^  ,  T  denotes  transpose. 

If  A  and  B  are  matrices  of  orders  a  and  p,  respectively, 

(c)  trace  (A  ®  B)  =  (trace  A)(trace  B) 

(d)  det(A  ®  B)  =  det(AP)det{B®) 

(e)  A  ®  B  =  (A  ®  ®  ®)  t  =  identity  matrix  of  order  a 

(f)  (A  ®  B)"^  =  A"^  ®  B’^ 

(g)  (L^  ®  Mj)(A  ®  BHL^  ®  M2)  =  A^  ®  Bj, 

where  1jj^AL2  =  A^  and  M^BM2  =  B^ 
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% 
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(h)  (L  O  M)‘V  ^  BHL  ®  M)  =  Aj  «  Bj, 

where  L'^AL  =  Aj  and  M’^BM  =  Bj^ 

POWER  PRODUCTS 

In  (A.  1),  let  the  two  transformations  be  identical.  Then  the 
(l/2)a(a+l)  distinct  products 

**  12**  *2 
^2 . ?2  '  ?2  ^ . 5a 

are  related  to  the  corresponding  unprimed  products  by  a  transformation 
whose  matrix  A...  is  called  the  second  power  matrix  of  A.  In  symbols 

x'  ‘X  x’  =  (A  -X  A)(x  ‘X  x) 

t 

*(2)  =  ^(2)*(2) 

where  x  =  (5^  ....  5^)  and  x'  =  (5^  .....  5^^  ). 

The  ith  power  product  of  x  is  similarly  'defined  as  the  vector  x^^j 
having  as  components  the  )  distinct  ith-degree  products  of 

components  of  x.  arranged  in  the  lexicographic  order,  x^^  and  x'^^ 
are  again  linearly  related: 

x!..  =  A,..x. 

(i)  (i)  1 

where  A...  is  the  matrix  of  the  linear  transformation. 

(i) 

The  lexicographic  ordering,  of  the  components  of  x,  is  merely  a 
convenient  one.  If  some  other  ordering  is  used,  the  rows  and  columns 
of  A^^^  will  be  correspondingly  permuted. 

The  Kronecker  and  power  products  have  several  similar  properties. 
In  particular,  the  following  theorems  hold: 

(a)  AB^.j  = 

If  A  is  of  order  a,  then 

(b)  det  A^.j  =  det(A'^)  , 
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t 


and 


(c)  has  as  eigenvalues  the 


'a  +  i  -  1 


products  of  the 


ith  degree  of  the  eigenvalues  of  A. 

APPENDIX  B 

A  MODIFIED  HOLDER  INEQUALITY 


Let  Z.,  j  =  1,  2,  .  .  .  ,  n,  be  scalar  random  variables.  Then 
i.  .  i./i 

En  I  zJ  I  <  n(E|z|  |)J  (B.i) 

j  j 


i.  >  0  for  all  j 
J 


PROOF:  In  the  case  of  two  random  variables  X,,  X_,  there  exists 

*>1  1  z 


the  well-known  Holder  Inequality^^ 

ElX^X^I  <(E|Xjl|)  MeIX^^I)  2 


(B.  2) 


—  +  “  =  1.  ^1*  ^2  ^  ° 
12  ^  ^ 


Under  the  substitutions  Xj  =  U^,  X^  =  V^,  l/r^  =  j/(j+k), 
l/r2  =  1  -  l/r^  =  k/(j+k),  (B.  2)  becomes 

E|U-'v^I  <  (E|  U'’'''^|  )  {E|  ,  (B.3) 


,k 


j,  k  >  0 

Let  V*'  =  Z^^  Z^**  ,  kj  +  k^  =  k,  kj,  k^  >  0.  Then 
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J 


1 


« 


t 


note  that  the  sum  of  the  exponents  of  and  is  (j+k). 
Upon  substituting  for  V,  (B.3  )  becomes 


(e|u-^zJ^Z2  <  (e|  z 


.•F 


) 


Now  apply  inequality  (B.3  )  to  the  extreme  right-hand  term  in  the 
above  inequality  to  obtain 

(e|zJ/*'| 

which  can  be  successively  generalized  to  (B.  1). 


APPENDIX  C 

SOME  THEOREMS  ON  ALMOST  SURE  AM)  ALMOST  UNIFORM- 
IN-«  TYPES  OF  STABILITY 

Theorems  1,  3  and  4,  relating  almost  sure  and  almost  uniform- 
in-w  types  of  stability  are  proven.  The  proofs  are  modeled  after  those 
of  Egoroff's  Theorem  (for  finite  measure,  convergence  almost  every¬ 
where  implies  almost  uniform  convergence)  and  its  "converse" 

(almost  uniform  convergence  implies  convergence  almost  everywhere). 

THEOREM  1:  The  trivial  solution  of  the  differential  system  (2.  2) 
is  stable  almost  surely  (A)  if,  and  (B)  only  if,  it  is  stable  almost  uni- 
formly-in-u. 

PROOF:  (A)  By  hypothesis,  given  any  positive  integer  n  and  any 
tp,  there  exists  an  u-set  B^(tjj)  with  PB^  >  1  -  l/n,  and  given  any 
c  >  0,  there  is  a  corresponding  5(c  ,  n,  t^)  >  0  such  that  ||  <6 

implies  that  ||  g(t,  x^,  t^;  u)  ||  <  c  for  all  i  f q  ^i^<i  ^ii 

Let  S  =  Q  B^.  Then  for  every  n,  SZDB^  and 
n=l  ” 

PS  >  PB  >  1  -  l/n.  Hence  PS  =  1. 

—  n 
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To  show  that  the  trivial  solution  is  stable  for  all  realizations 
corresponding  to  the  points  of  S  =  S(tQ),  choose  any  a>' t  S.  Then 
b)'  4  for  some  k  =  k(co')  whence  for  any  t^  and  any  c  >  0  , 

||xq1|  <6{€,  k(aJ'),tQ)  implies  that  ||  g(t,  Xq,  tQ:«')  ||  <€  for  all  t  >  t^. 

(  t 

Given  any  other  initial  time  t^  ,  S(tQ )  can  be  taken  to  be  S(tQ). 
For,  if  the  trivial  solution  of  the  deterministic  system  corresponding 

I 

to  <^eS(tQ)  is  stable  at  t^,  it  is  stable  at  t^  that  is,  given  any  c  >  0, 

there  is  a  6(€  ,  t^  ,<»> )  >  0  such  that  ||  x^  ||  <  6(e  ,  t^  ,w  )  implies 
II  g(t,  Xq,  tg  ;w  )  II  <  €  for  all  t  >  tp  .  Hence  there  is  a  set  S  of  unit  pro 
bability  on  which  the  trivial  solution  is  stable  at  every  t^.  Therefore 
the  trivial  solution  is  stable  almost  surely. 

(B)  by  hypothesis,  there  exists  an  u-set  S  of  unit  probability, 
and  to  every  w  t  S,  every  positive  integer  m,  and  every  t^,  there 
corresponds  a  positive  integer  n  (m.w.tQ)  such  that  ||  Xq||  <  l/n 
implies  ||  g(t,  x^,  tQ;w)  jj  <  l/m  for  all  t  >  t^. 

Given  any  t^,  let 


= 


(w: 


- ^11  *Q* 


for  all  t  >  tp)  ,  m,  n  =  1,  Z,  . 


If 


u  t  E 


w  e  S.  then  for  every  m  there  is  an  nj(m,  w,  t^)  >  0  such  that 

I  rni _ r _  I  I  ~  i  I  I 


Therefore  H  E  ZD  S,  whence  P  IJ  E  =1.  Since 
n  n 

n  n 


_m._  _m,_  . .  m  ,  .  «  , 

E,  CZ.E,  CUE,  C_.  . .  ,  lim  PE  =  1  and  given  any  ii  >  0  there 
1  Z  J  n  ' 

exists  a  smallest  integer  Ntm.^.t^)  such  that  >1  -  ii/z”^. 

Define  B  =  Q  E^  .  Note  that  B  depends  on  and  t^  only. 


m  m 


11  • 


Hence  PB  >  1  -  t].  Moreover,  for  the  t^  chosen  above,  if  u'  £  B,  then 
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for  all  m  so  that  j|  Xq||  <  l/N(m,ii,tQ)  implies 

II  g(t,  Xq,  tQ;u')  II  <l/m  for  all  m  and  for  all  t  >  t^.  Therefore  the  trivial 
solution  is  stable  almost  uniformly- in . 

t 

THEOREM  3  :  The  trivial  solution  of  (2.  2)  is  quasi-asymptotically 
stable  almost  surely  if  it  is  quasi-asymptotically  stable  almost  uni¬ 
formly -in- u  with  the  set  B  (see  definition  (iii-f),  Sec.  II)  independent 
of  Xq. 

PROOF:  By  hypothesis,  given  any  integer  n  >  0  and  any  t^,  there 

is  a  6(n,  t.)  >  0  and  an  u  -set  B  (t.)  with  PB  >  1  -  l/n  such  that 
'  ’  0  n'  0  n 

II  XqII  <6  implies  that  g(t,  x^,  )  — >0  as  t — >oo  uniformly  on  B^. 

Let  S(tQ)  =  y  Then  for  all  n,  and 

PS(t-)  >  PB  (t.)  >  1  -  l/n.  Hence  PS(t-)  =  1. 

0  —  n  u  u 

Given  any  u  e  S(tQ),  u  is  an  element  of  B^(tQ)  for  some  k  =  k(b)). 
Hence  at  t^,  ||  Xq||  <  5(k(u)),  t^)  implies  g(t,  x^,  tQ;w)->0  as  t— ^oo; 
i.  e.  ,  the  trivial  solution  is  quasi-asymptotically  stable  almost  surely 
,  at  t^.  As  in  the  proof  of  Theorem  1,  S  can  be  taken  to  be  independent 

of  t^.  This  completes  the  proof. 

THEOREM  4:  The  trivial  solution  of  (2.  2)  is  quasi-asymptotically 
stable  almost  uniformly- in-u>  if  it  is  quasi-asymptotically  stable 
almost  surely. 

PROOF:  By  hypothesis,  there  exists  an  u-set  S  of  unit  probability 
auid  to  every  u  &  S  and  any  given  t^,  there  corresponds  a  positive  integer 
n  (  «,  tp)  such  that  ||  Xq  ||  <  l/n  implies  g(t,  x^,  )  — >0  as  t— >  oo. 

Given  any  t^,  let 

^n<‘0^  =  ^n  =  ^“=11  *oll  <  *“>«>}. 

A  s  2p  t  •  t 

Pick  u  e  S.  Then  u  a  F— ,  .  .  and  hence  u  a  U  F  .  Therefore 

n(  w,  tp)  n  n 

*  Uf  ZDS  whence  pU  F  =  1.  Since  F.C  F.C  F,  . . .  ,  lim  PF  =1. 

nn  nn  123’  n 

n 

Hence,  given  any  tj  >  0  there  is  a  smallest  integer  N(il,tQ)  such  that 
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t  PFj^  >  1  -  Ti/2. 

For  every  Xq,  ||*q!I  <1/N,  define 

'  =  II  *0’  ^0’“^  II  "^41  all  t  >  tp  +  k  } 

m,  k  s  1,  2,  ... 

If  u  fcFj^,  then  for  every  m  there  is  a  k^(m,  w,  x^,  t^)  such  that 
u)  £,  .  Therefore  ^  ^3  Fj^,  whence  P  e|^  >  1  -  ii/2. 

Since  E^  CZE^CE^G.,  ...  ...  Jim  PE^  >1  -  r[ll  and  there  exists  a 

smallest  integer  K(m,  Xq,  t^)  >  0  such  that  lim  PE^  -  PE^  <  (r\IZ)l‘^ 

Ic 

from  which  it  follows  that  PE^  >  1  -  (tj/2)(1  +  l/z”'). 

Define  B  =  Q  E^  and  note  that  B  =  B(ii,  Xq,  t^). 

pb'  =  p  (E”)'<  2  P(Ek  ''i  I  >>  =  ’' 

m  m 

Hence  PB  >  1  -  tj.  Moreover,  if  u>'  £  B,  then  w'  £  E^  for  every  m  so 
that  II  XqII  <  l/N(Ti,tQ)  implies  ||  g(t,  x^,  t^;  w*)  ||  <  l/m  for  all  m  and  for 
*  all  t  >  tQ  +  K(rri,  T,  Xq,  tp).  Therefore  the  trivial  solution  is  quasi- 

asymptotically  stable  almost  uniformly- in-  w  . 

APPENDIX  D 

E(5k§  k-1'  *  •  5  1^  matrix  VALUED  MARKOV  CHAIN 

Given  a  finite  Markov  chain  with  initial  probability  vector 
(a, ,  .  .  .  ,  a  )  and  an  m  x  m  constant  matrix  P  =  (p..)  of  transition 
probabilities,  associate  with  each  state  i  an  n  x  n  matrix,  X^.  Let 
be  the  random  matrix  occurring  at  time  k,  k  =  1,  2,  ....  The 
^  expected  value  of  the  product  matrix  ik  1’  *  1  derived  here. 

For  notational  and  algebraic  simplicity  consider  a  four-fold 
product.  Then 
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E 1453 12  = 


1  (  Z  ^jPyPjk)  ^i^Oki  <°  '> 


i.  k,l 

Let  I  be  the  identity  matrix  of  order  n  and  define  Q  as  the  Kronecker 
product 


Q  =  P  ®  I 

and  Y  as  the  direct  sum 

y  =  Xj®X2©...  ox 

Then 

T  T 
U  =  Q^YQ 


m 


(D.  2) 


i 

■pjjl  Pjjl 

0 

t _ 

^11^  ^21^  ’  ■  ■ 

II 

•“iz*  ■’22^ 

0  X2  . . . 

Pl2^  ^22^  *  *  • 

■  ■  ■_ 

... 

^21^2^12  ^11^1*^21  ^21^2^22 

^12^1^11  ^  ^22^2^12  Pl7^lP71  ^77^7^77 


12"1‘'21  "  *'22''2‘'22 


["ikl 


(D.3) 


where  is  the  matrix  entry  in  row  i  and  column  k.  The  parenthe* 
sized  term  on  the  right  side  of  Eq.  (D.  1)  can  be  identified  as  so 
that  (D.  1)  becomes 
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i 
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By  steps  similar  to  (D.  2)  through  (D.  3), 

I  ^'k^ki'-W  = 

k 

where 

V  =  [V.^]  =  q'^yu 
Equation  (O.  4)  now  reduces  to 

Ei4i3l2i,  =  '"liVi 

I  i 


Let  ^  be  the  Kronecker  product  of  the  initial  probability 
the  identity: 


a  - 


9  I 


Let  W-  =  /  V.  .X.a.  auid  let  W  be  the  matrix 

I  Cj  5i  1  1 


W  = 


w. 


ro 


=VYCi 


Equation  (D.  6)  now  becomes 


YW 


(D.4) 


(D.5) 


{D.6) 

vector  with 


(D.  7) 


(D.8) 
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where 


1 


t 


i  =  (I.  I) 

m  terms 

Substituting  successively  for  W,  V,  U  by  using  (D.  7),  (D.  5),  (D.  2) 
respectively,  (D,  8)  reduces  to 

Ei4i3i2li  = 

which  can  be  successively  generalized  to 

APPENDIX  E 

SPECIALIZATION  OF  MARKOV  CASE  TO  INDEPENDENT  CASE 


In  ChapJII,  Sec.  3.  2,  it  was  shown  that  if 


*k  ■  ^  1*0 

with  {  J,  }  an  independent,  identically  distributed,  matrix  sequence, 
^  k 

then  EXj^  ^ .  j  =  ^^5  [i]^  *0[i]  ’  ^  appendix,  this  result  is  derived 
as  a  special  case  of  Eq.  (4.  8}; 

.T,kiL 


E*k[i]  =  ^i^^A  >  ^^i^i*0[i] 


which  is  the  result  when  {  i®  Markov  chain. 

If  {  }  is  an  independent  proces«,  then  the  Markov  matrix 

has  Identical  rows:  p..  =  p,  =  P[  J  v  “  Hence  T.Q. 

IJ  J  K  j  11 

has  the  form 
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T 

= 


L  ^m^m[  i]  ^m^m[  i] 


Pi^i[i] 


P  X  r.i 
'^m  m[  ij 


Also  a.  =  p.,  whence 
1 


^i^i  = 


Therefore, 


pi’Sti] 


P  X  r.. 


y.qTy.q,. 

1111 


PlXi[ijEi[.] 


^m^m[  i]^  ^  [  i] 


and 


and,  in  general. 


2  -1 


Pi^i[i] 


LPni^m[i]<^5[i])  J 


Pi^k[i] 


k-r 


[i]^ 


k-1 
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where 


I 


J 


\ 


I 


j 


Equation  (4.8)  now  reduces  to 

which  is  the  desired  result. 


APPENDIX  F 

COMPUTATION  OF  X.(Y.Q^)  FOR  CYCLIC  CHAINS 

Suppose  that  the  Markov  chain  is  ergodic;  that  is,  there  is  only 
one  ergodic  set  and  the  transient  set  is  empty.  Suppose  further  that 
the  ergodic  set  is  cyclic,  of  period  d.  The  modes  are  assumed  to  be 
suitably  ordered  so  that  the  transition  matrix  E  has  the  form  (4.  4); 


E  =: 


0 

0 


0 

0 


0 

c. 

i 

0 

0 


Hence  the  Y  matrix  has  the  form 


0 

0 


'd-1 


Y.  = 

1  1 


«  x^' 


where  x!"^^  ’ 


is  the  direct  sum  of  the  matrices  X 


Ifi] 


corresponding 


to  the  modes  I  £ 

J 
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1 


®  I„i ) 

can  now  be  written  as 

“0  0  ...  0  F^- 

Fj  0  ...  0  0 

0  F2  • • •  0  0 

.0  0  ...  F^_j  0  . 

where 


T 

YiQi  = 


F  =  (C^  ®  I  )  ,  j  =  1 . d. 

J  ^  J  n^ 

provided  is  interpreted  as  x!^^  . 

T 

Raising  to  the  dth  power, 


2  ®  ^Z^l^d^d-l' 


•  ^3  ®  •  •  •  ®  ^d^d-r  •  ■  ^2^1- 


All  of  the  cyclic  products  of  the  F.'s  on  the  right-hand  side  of  the 

J 

above  equation  are  square,  but  not  necessarily  of  the  same  order. 

Moreover,  by  Lemma  1  proven  below,  all  of  the  cyclic  products  have 

the  same  eigenvalues,  except  possibly  for  the  zero  eigenvalue.  Hence 

T 

the  nonzero  eigenvalues  of  Y.Q.  are  the  nonzero  eigenvalues  of  any 

X  X 

of  the  cyclic  products.  Since  only  the  nonzero  eigenvalues  of  Y^Q^ 
are  of  interest,  a  convenient  way  to  compute  them  is  to  compute  the 
eigenvalues  of  the  lowest  order  cyclic  product. 


n. 

J 


LEMMA  1:  Let  A^, 

X  n.  ,  with  n,  =  n, 
j+1  1  k+1 


=  1,  .  .  .  ,  k,  be  matrices  of  dimension 
Then  all  the  cyclic  products  ^^2 
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AjAj. 


.  .  Aj^  1  have  the  same  eigenvalues, 


except  possibly  for  the  zero  eigenvalue. 
The  proof  follows  from 


LEMMA  2:  Let  A  be  an  r  x  s  matrix  and  let  B  be  an  s  x  r 
matrix.  Assume,  for  convenience,  that  r  >  s.  Then, 

(a)  AB  and  BA  are  of  dimension  r  x  r  and  s  x  s  respectively; 

(b)  zero  is  an  eigenvalue  of  AB  of  multiplicity  at  least  (r-s); 

(c)  every  eigenvalue  of  BA  is  an  eigenvalue  of  AB. 

PROOF:  (a)  Follows  from  the  definition  of  matrix  multiplication, 
(b)  Let  N  be  the  r  x  (r-s)  matrix  all  of  whose  elements  are  zero. 
Define  X,  S  to  be  the  matrices 


X  =  [A,N] 


Then 


"X  B  =  AB 


By  inspection,  zero  is  an  eigenvalue,  of  multiplicity  (r-s),  of  B  X. 
But  the  products  X'B,  'B  X  have  the  same  eigenvalues  since  X  and  B 
are  square  matrices.  Therefore,  zero  is  an  eigenvalue,  of  multi¬ 
plicity  (r-s),  of  AB  (=  X'B),  and 

(c)  every  eigenvalue  of  BA  is  an  eigenvalue  of  AB. 
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